In short. The longest matching robots.txt rule wins, with Allow preferred for equivalent Allow and Disallow rules.
Definition: In robots.txt, a file of crawl rules, the longest matching rule determines whether a crawler may access a URL. RFC 9309 requires the most specific match, measured in octets (bytes), and recommends Allow when an Allow rule and a Disallow rule are equivalent.
Key facts
- RFC 9309, section 2.2.2 requires matching to begin at the start of the URL path and use the match with the most octets.
- As of 17 October 2026, Google's robots.txt specification uses path length, then Allow to resolve equal-length conflicts, including wildcard rules.
- Google's guide to writing robots.txt specifies case-sensitive rules:
/file.aspand/FILE.aspdiffer.
What does RFC 9309 require?
RFC 9309, published in September 2022, describes comparison against the rules of the applicable user-agent group. A group is a set of rules introduced by a User-agent line identifying the crawler. Matching groups are combined before their path rules are evaluated.
Section 2.2.2 requires the most specific matching path. A shorter Allow can therefore lose to a longer Disallow, and a longer Allow can override a broader Disallow. Moving a path rule earlier or later within the applicable group leaves that comparison unchanged.
The standard uses MUST for longest-match selection and SHOULD for preferring Allow when the opposing rules are equivalent. If none of the applicable rules matches a URL, access is allowed.
How does Google resolve concrete conflicts?
Google's specification examples show these rules for https://example.com/page:
allow: /p
disallow: /
The Allow path is longer, so Google permits crawling. The same documentation gives this pair for https://example.com/folder/page:
allow: /folder
disallow: /folder
Both paths match equally. Google chooses Allow as the least restrictive rule.
A wildcard is a pattern character: * matches any sequence of characters, including an empty sequence. Google's example for https://example.com/page.htm uses:
allow: /page
disallow: /*.htm
Google chooses Disallow because that matching rule path is longer. Its example for https://example.com/page.php5 instead pairs allow: /page with disallow: /*.ph; Google chooses Allow for the equal-length conflict. The expanded text matched by * does not itself give that rule priority.
How can a website check the result?
Google's robots.txt writing guide describes groups headed by User-agent and combining groups for the same user agent. Checking a conflict starts with the rules applicable to the intended crawler. Path lengths from unrelated groups do not compete.
The guide also describes opening the site's /robots.txt in a private browser window to confirm that the uploaded file is publicly accessible. For local testing, it points to Google's open source robots.txt library, which Google also uses in Search. Testing the full file against the exact URL preserves the relevant group and path matching context.
Related terms
- robots.txt
- robots.txt Allow directive
- robots.txt Disallow directive
- robots.txt wildcard matching
- robots.txt group precedence
FAQ
Does the last matching line win?
No. RFC 9309 selects the most specific match by octet length. Reordering Allow and Disallow within the same applicable group preserves that rule.
Does Allow always win?
No. Google's specification shows Disallow winning for /page.htm when /*.htm competes with /page. Allow wins the equivalent-path tie in its /folder example.
Sources
- IETF, "RFC 9309: Robots Exclusion Protocol", September 2022
- Google, "How Google interprets the robots.txt specification", updated 31 August 2026
- Google, "How to write and submit a robots.txt file", updated 21 November 2025
Check how a page reads to answer engines. Open the GEO score analyzer