In short. RFC 9309 requires matching crawler groups to merge and uses wildcard rules as a fallback when no named group matches.

Definition: robots.txt group precedence determines which crawler-specific set of rules applies to a request. RFC 9309 requires matching groups to be combined and makes the wildcard group a fallback when no named group matches.

Key facts

RFC 9309, section 2.2.1 requires case-insensitive matching of crawler names and merging of matching groups.
As of 17 October 2026, Google's robots.txt specification states that its crawlers select the most specific matching user agent, regardless of group order.
Google publishes a robots.txt parser and matcher with a binary for testing a single URL and user agent against a file.

How does RFC 9309 select groups?

A group contains one or more User-agent lines followed by rules. A product token is the crawler name used to find a relevant group, such as ExampleBot. RFC 9309 requires case-insensitive matching, so changing that name's capitalization preserves the match.

When several groups match, their rules form a combined group. The RFC illustrates this with separate ExampleBot groups: exclusions for /foo and /bar in one group combine with an exclusion for /baz in another. A later matching group therefore adds rules to the applicable set.

The wildcard value * supplies fallback rules. A crawler with a matching named group uses that group's rules; fallback applies when the named match is absent.

How does Google document selection and merging?

Google's specification gives this example:

user-agent: googlebot-news
disallow: /fish

user-agent: *
disallow: /carrots

user-agent: googlebot-news
disallow: /shrimp

Google merges the googlebot-news groups into a group containing both /fish and /shrimp exclusions. The /carrots exclusion remains in the global group. To apply that restriction to Googlebot News as well, the named group's rules must include it.

The same documentation describes selection when googlebot-news, googlebot, and * groups coexist: Googlebot News follows the googlebot-news group, while Googlebot for web search follows googlebot. Placing the wildcard group between them leaves those selections unchanged. Google's most-specific-user-agent rule and merging of repeated applicable groups operate together.

Which rule applies after groups are merged?

Allow permits access to a matching path; Disallow excludes it. RFC 9309, section 2.2.2 requires the most specific matching path, measured in octets, or bytes, to determine access. Equivalent Allow and Disallow rules should resolve to Allow. Duplicate rules may be removed.

This path comparison takes place within the applicable combined group. The position of an exclusion in a repeated group gives it no extra priority. If no path rule matches, access to the URI is allowed.

How can a website check a combined rule set?

Google's parser library README describes a binary that tests a local robots.txt file with a supplied user agent and URL. Testing a path covered by each repeated group helps check the combined restrictions. The README also states that the library omits crawler logic outside parsing and matching, including some crawler-specific fallback behavior.

FAQ

Can I place a repeated crawler group at the end to replace earlier rules?

Under RFC 9309, matching groups merge. To remove an exclusion, you must remove or change the applicable rule rather than rely on the later group's position.

Does a blank line separate groups?

A blank line can appear inside a group. RFC 9309's group syntax allows empty lines between user-agent lines and between rules; a new user-agent line after rules starts another group.

Sources

Check how a page reads to answer engines. Open the GEO score analyzer