In short. Faceted navigation filters collections, while URL rules determine how many filter combinations crawlers can request.

Definition: Faceted navigation lets visitors filter a collection by item attributes. When filters create separate URLs, their combinations form a crawl space: the set of URLs available for crawlers to request.

Key facts

As of 17 October 2026, Google's faceted navigation documentation recommends preventing crawling of filter URLs that have no intended search visibility.
Google's crawl budget guide advises against using noindex, an indexing exclusion instruction, to save crawling time because Google still requests the page.
Google's robots.txt introduction explains that this file controls crawler access, while a blocked URL can still appear in search results when linked elsewhere.
Google's noindex documentation requires crawler access so Google can read an indexing exclusion.

Why do filter URLs multiply?

URL parameters are named values in the query string after ?. Google's faceted navigation example uses these parameters to select fish by color and size:

https://example.com/items.shtm?products=fish&color=radioactive_green&size=tiny

Changing a value changes the selection. Many combinations can therefore create many URLs for the underlying collection.

A crawl budget is the time and resources a crawler allocates to a site. Google's crawl budget guide explains that duplicate or unimportant URLs can consume this allocation. Filter design therefore affects the volume of pages a server must serve to crawlers.

How can unnecessary filter crawling be limited?

A robots.txt file declares which URLs a crawler may access. Google's faceted navigation guidance gives this example for Googlebot, Google's search crawler:

user-agent: Googlebot
disallow: /*?*products=
disallow: /*?*color=
disallow: /*?*size=
allow: /*?products=all$

The rules block the named filter parameters and allow the specified unfiltered listing. Google's example preserves crawling of individual item pages alongside that listing.

A canonical link identifies a preferred version of a page. The same guidance says rel="canonical" may reduce crawling of alternative filter URLs over time, while describing it as less effective for crawl control than robots.txt or fragment-based filters. A fragment is the part after #; Google generally excludes fragments from crawling and indexing.

What if filtered pages need search visibility?

Google's filter URL recommendations specify & as the parameter separator. For filters encoded in paths, the logical order must stay consistent and duplicate filters must be prevented.

The same documentation calls for HTTP 404, the response status for a missing page, when combinations return no results. It also applies this response to invalid combinations and nonexistent pagination URLs. Google specifies serving the error at the requested URL, without redirecting empty combinations to a shared error page.

How can crawl control and index exclusion be checked?

noindex is an instruction to exclude content from search indexing, supplied through a page's HTML or an HTTP response header. Google's implementation guide recommends the URL Inspection tool to check the HTML Googlebot receives. Search Console's Page Indexing report shows pages where Googlebot extracts this instruction.

Google's robots.txt documentation distinguishes access control from exclusion: an externally linked, blocked URL may still appear in results. The noindex guide therefore requires the page to remain accessible for the instruction to take effect.

FAQ

Should every filter URL remain crawlable?

Google's faceted navigation guide bases the choice on whether filtered pages need potential indexing. You can exclude unnecessary combinations while preserving access to item pages and an unfiltered listing.

Can I combine a robots.txt block with noindex?

Google's noindex guide requires access to the page to read the instruction. If you need Google to process noindex, leave the page accessible to Googlebot.

Does blocking filters automatically increase crawling elsewhere?

Google's crawl budget guide says it reallocates the freed capacity only when crawling already reaches the site's crawl capacity limit, the limit Google sets to avoid overloading the server.

Sources

Check how a page reads to answer engines. Open the GEO score analyzer