In short. The robots.txt User-agent directive matches crawler product tokens to groups of crawling rules, with an asterisk group providing a fallback.

Definition: In RFC 9309, the robots.txt User-agent directive identifies the crawler product token to which a group of crawling rules applies. A product token is the crawler's chosen name; a group contains User-agent lines followed by Allow or Disallow rules for URL paths.

Key facts

RFC 9309 requires case-insensitive matching between a crawler's product token and the User-agent value.
As of 17 October 2026, Google's robots.txt specification says that its crawlers combine matching specific groups and keep them separate from the global * group.
Google's creation guide recommends checking public access in a private browser window and offers its robots.txt library for local markup tests.

Which name belongs after User-agent?

RFC 9309's User-Agent section defines product tokens using letters, underscores and hyphens. It recommends that the token appear as a substring of the crawler's identification string, such as an HTTP User-Agent request header. Its example uses ExampleBot as the robots.txt token within a longer header identifying that crawler.

The substring relationship concerns the token's appearance in the request header. The robots.txt group is matched against the product token. Letter case has no effect on that match.

How are groups selected and combined?

Under RFC 9309, consecutive User-agent lines before the rules share a group. Once rules begin, a subsequent User-agent line starts another group. Blank lines can occur inside groups.

This example follows the RFC's illustration of repeated groups:

User-agent: ExampleBot
Disallow: /foo
Disallow: /bar

User-agent: ExampleBot
Disallow: /baz

The matching crawler combines these groups, so /foo, /bar and /baz all remain disallowed. If no named group matches, the crawler follows User-agent: *, when present. If neither matches, no rules apply.

After group selection, the crawler evaluates Allow and Disallow path rules. The RFC requires the matching path with the most octets, meaning bytes, to take precedence.

What matching behavior does Google document?

Google's specification selects the most specific matching user agent. In its example, googlebot-news selects its own group when the file also contains googlebot and * groups. Googlebot for web crawling selects googlebot.

The same documentation says group order is irrelevant. Moving the global group above or below a specific group therefore leaves Google's selection unchanged. Repeating a specific User-agent value combines its rules. Adding a specific group can consequently change which restrictions apply, because Google's crawlers exclude the global group's rules from that combination.

How can selection be checked?

Google's creation guide describes opening the public /robots.txt URL in a private browser window before testing markup. It identifies the robots.txt report in Search Console for accessible files and Google's open source robots.txt library for local tests.

For a file with repeated groups, a local test should cover paths from each matching group. For a file with specific and global groups, checking a path restricted only in the global group helps reveal which rules the parser selects.

FAQ

Yes. Google's specification shows consecutive User-agent lines sharing the rules that follow them. Place the names before those rules.

Does a blank line end a group?

RFC 9309's grammar permits blank lines within a group. After the rules begin, a new User-agent line or the end of the file ends that group.

Sources

Check how a page reads to answer engines. Open the GEO score analyzer