Can robots.txt rules match URL query strings?

Definition: As of October 2026, Google's robots.txt rules can match URL query strings using allow and disallow patterns. A query string is the URL component introduced by ?.

Key facts

• Google's robots.txt specification defines * as matching zero or more valid characters and $ as the end of the URL.
• RFC 3986, section 3.4 defines the query as a separate component, ending at # or the end of the URL.
• Google's faceted navigation guide explicitly uses disallow: /*?*color= to restrict crawling of filtered URLs.

Why does Google call this path matching?

Google's specification calls its comparison "URL matching based on path values". Its examples include query strings: /fish matches /fish.php?id=anything, while /*.php$ excludes /filename.php?parameters from its matches.

In the terminology of RFC 3986, sections 3.3 and 3.4, the path ends where the query begins. The query contains data that helps identify a resource together with the path. The standard also permits / and ? within the query itself. These component definitions explain URL structure; Google's examples establish how its robots.txt matching handles that structure.

How can a rule target filter parameters?

Faceted navigation lets visitors filter displayed items. Google's faceted navigation guide gives this URL:

https://example.com/items.shtm?products=fish&color=radioactive_green&size=tiny

Changing products, color or size changes the displayed items. Filter combinations can create a very large set of URLs and consume server resources. For filtered URLs that need no search presence, the guide provides this configuration:

user-agent: Googlebot
disallow: /*?*products=
disallow: /*?*color=
disallow: /*?*size=
allow: /*?products=all$

The patterns place a literal ? before the parameter text. The wildcard after it accommodates preceding query text. Applying these patterns, color= can therefore match after another parameter in the example URL. The allow pattern ends at products=all; adding further parameters removes that particular match.

The same guide recommends allowing individual item pages and an unfiltered listing page when filtered listings are blocked. It recommends & as the parameter separator for URLs intended to remain crawlable.

How can matching be checked against examples?

Google's specification supplies comparison cases. The following are separate rule alternatives:

disallow: /*.php
disallow: /*.php$

The unanchored pattern matches /folder/filename.php?parameters. The anchored pattern matches /filename.php and excludes /filename.php?parameters. Path matching is case-sensitive: /windows.PHP falls outside both patterns.

Related terms

• robots.txt
• URL parameters
• Faceted navigation
• robots.txt group precedence

FAQ

Does ? match any single character?

Google's specification lists * and $ as wildcard characters. In the query patterns, ? is literal.

Does blocking a query URL remove it from Google Search?

Google's specification says a blocked URL can appear in search results without a snippet. Robots.txt controls crawling.

Sources

1. Google, "How Google interprets the robots.txt specification", updated 31 August 2026
2. IETF, "Uniform Resource Identifier (URI): Generic Syntax", January 2005
3. Google, "Managing crawling of faceted navigation URLs", updated 18 December 2025

---

Check how a page reads to answer engines. Open the GEO score analyzer