Does a bot-specific robots group inherit wildcard rules?

Definition: A robots.txt group pairs crawler names with crawl rules; a bot-specific group does not inherit rules from a separate User-agent: * group. Under RFC 9309, the wildcard group supplies fallback rules when no group matches the crawler's product token, the name it uses to identify its applicable rules.

Key facts

• RFC 9309 requires case-insensitive matching of product tokens and merging of groups that match the same token.
• As of 17 October 2026, Google's robots.txt specification explicitly states that user-agent-specific groups and global wildcard groups are not combined.
• Google's file creation guide permits multiple User-agent lines in a group, allowing named crawlers to share the rules that follow.

How does the fallback work?

A group consists of User-agent lines followed by rules such as Disallow, which excludes matching paths from crawling, and Allow, which permits them. RFC 9309's group selection rules first match the crawler's product token. Matching groups are combined. The wildcard group applies if no matching group exists.

The fallback concerns group selection. A path absent from a bot-specific group's rules does not trigger a second lookup in the wildcard group. The same RFC states that a URL is allowed when none of the rules in the applicable group matches it. If neither a matching group nor a wildcard group exists, no rules apply.

What does Google's example show?

Google's grouping example contains these groups:

user-agent: googlebot-news
disallow: /fish

user-agent: *
disallow: /carrots

user-agent: googlebot-news
disallow: /shrimp

Google merges the googlebot-news groups into a rule set excluding /fish and /shrimp. The separate /carrots exclusion remains in the wildcard group. For Googlebot News, /carrots is therefore allowed by this file, because its applicable rules contain no matching exclusion.

The same specification says group order is irrelevant. Moving the wildcard group above the named groups leaves their meaning unchanged. Google selects the most specific matching user agent; its documented example selects googlebot-news for Googlebot News even when a googlebot group also exists.

How can the common exclusion be preserved?

Following Google's documented separation of specific and global groups, an exclusion intended for the named crawler must appear in its applicable rules. This adaptation of Google's example also excludes /carrots for Googlebot News:

user-agent: *
disallow: /carrots

user-agent: googlebot-news
disallow: /fish
disallow: /shrimp
disallow: /carrots

The repeated line makes the intended policy explicit. Reviewing a newly added bot-specific group therefore includes checking which wildcard exclusions still need to apply to that bot. Path matching then operates within the selected rules: the RFC requires the most specific matching path rule to be used.

Related terms

• robots.txt
• robots.txt User-agent directive
• robots.txt group precedence
• robots.txt Disallow directive

FAQ

Must I repeat every wildcard exclusion for a named bot?

You must include each exclusion intended for that bot in its applicable group or matching groups. RFC 9309 makes the wildcard group a fallback, so exclusions left only there do not carry over.

Does the first matching group win?

Google documents selection of the most specific matching user agent and merging of repeated groups for that user agent. File order does not determine which of those groups takes effect.

Does an Allow rule compete with a wildcard Disallow?

Rules compete within the applicable group after group selection and merging. Google's separate wildcard group contributes no path rules when a specific group applies.

Sources

1. IETF, "RFC 9309: Robots Exclusion Protocol", September 2022
2. Google, "How Google interprets the robots.txt specification", updated 31 August 2026
3. Google, "Create and submit a robots.txt file", updated 21 November 2025

---

Check how a page reads to answer engines. Open the GEO score analyzer