In short. In robots.txt path patterns, * matches zero or more characters and $ requires the match to reach the end of the URL.

Definition: Under RFC 9309, robots.txt wildcard matching compares Allow and Disallow path patterns with requested URLs. The asterisk (*) matches zero or more characters, and the dollar sign ($) anchors a match to the end of the URL.

Key facts

RFC 9309 requires crawlers to support * and $ in path patterns and to begin matching at the first byte of the path.
As of 17 October 2026, Google's robots.txt specification treats /fish* and /fish as equivalent: a trailing wildcard adds no restriction.
Google's file creation guide documents case-sensitive rules: Disallow: /file.asp applies to /file.asp and leaves /FILE.asp outside that match.

How does the wildcard match a path?

The special-character rules in RFC 9309 let * consume any sequence of characters, including an empty sequence. Its simple example explicitly includes the forward slash among the characters the wildcard can match. A wildcard can therefore span directory boundaries.

Matching begins at the start of the URL path. Without an end anchor, reaching the end of the rule pattern is sufficient for a match, even when the requested URL continues. Google's path examples show /fish matching /fishheads and /fish/salmon.html. Adding a slash changes the literal prefix: /fish/ matches /fish/salmon.htm, while /fish itself falls outside that pattern.

What does the end anchor change?

Google's matching table distinguishes these alternative rules:

Disallow: /*.php
Disallow: /*.php$

The unanchored pattern matches /folder/filename.php?parameters and /folder/any.php.file.html. The anchored pattern matches /filename.php and /folder/filename.php, but excludes /filename.php?parameters and /filename.php/ from its matches. A query string, the part introduced by ?, therefore matters when checking an end-anchored rule.

These are patterns applied to URL text. The same table excludes / from /*.php even when the server maps that root URL to an index.php file. A server's internal filename does not determine the match.

Which rule wins when patterns overlap?

RFC 9309's matching rules require the most specific match, measured in bytes. Where an Allow rule and a Disallow rule are equivalent, the standard recommends using Allow. An unmatched URL is allowed within the applicable crawler group.

Google's precedence examples include:

Allow: /$
Disallow: /

For the root URL, Google selects Allow: /$ as more specific. For /page.htm, only the Disallow rule applies. These lines form an exception for the root within an otherwise disallowed site.

FAQ

How can you match a literal asterisk or dollar sign?

RFC 9309's encoding examples use /path/file-with-a-%2A.html to match a filename containing a literal *. The pattern /path/foo-%24 matches a path ending in a literal $.

How can you check a robots.txt file before relying on its patterns?

Google's testing instructions describe building its open source robots.txt library to test files locally. For a deployed file, you can also use the robots.txt report in Search Console to identify markup issues.

Sources

Check how a page reads to answer engines. Open the GEO score analyzer