robots.txt Allow directive

Definition: RFC 9309 defines Allow as a robots.txt rule permitting crawlers to access matching URL paths. An Allow rule can create an exception within a broader Disallow rule when its path is more specific.

Key facts

• RFC 9309 requires the most specific matching rule, measured in octets, and recommends Allow when Allow and Disallow rules are equivalent.
• As of 17 October 2026, Google's robots.txt specification documents case-sensitive path values and ignores an Allow rule with no path.
• Google's robots.txt introduction warns that blocking resources needed to understand a page can impair its analysis of that page.

How does Allow create an exception?

A robots.txt group consists of User-agent lines identifying crawlers, followed by rules. RFC 9309, section 2.2.2 requires path matching to begin at the first octet, meaning the first byte. This illustrative group applies its exceptions through progressively longer paths:

User-agent: *
Disallow: /docs/
Allow: /docs/public/
Disallow: /docs/public/drafts/

Here, /docs/public/guide.html is allowed because /docs/public/ is more specific than /docs/. The longer Disallow blocks /docs/public/drafts/outline.html. A short Allow therefore cannot override every Disallow in its group. Moving the lines within this group leaves their relative specificity unchanged.

The same section recommends choosing Allow when the rules are equivalent, such as Allow and Disallow both specifying /docs/.

Which crawler group contains the exception?

Google's group-selection rules select the group with the most specific matching user agent. Groups specific to a crawler are combined when repeated; a specific group and the global User-agent: * group remain separate.

An exception must therefore appear in the rules applicable to the intended crawler. Adding Allow to the wildcard group leaves the rules of an existing Googlebot-specific group unchanged. Group selection happens before the crawler resolves competing paths.

The same documentation distinguishes the field name from its value: Allow and allow are equivalent field names, while the path value is case-sensitive. Reviewing an exception means checking both its group and the exact capitalization of its path.

What does an exception change for a website?

Google's robots.txt introduction explains that scripts and styles can affect how its crawler understands a page. An Allow exception can keep a required resource accessible within a directory otherwise excluded from crawling. The resource's actual path determines whether the exception matches.

The same introduction separates crawling from search-result removal: Google can index a blocked page's URL through links elsewhere. It recommends noindex or password protection when the objective is to keep a web page out of Google.

Related terms

• robots.txt
• robots.txt Disallow directive
• robots.txt User-agent directive
• robots.txt group precedence

FAQ

Do I need an Allow rule for every public page?

Under RFC 9309, a URL is allowed when no rule in the applicable group matches it.

Does a blank Allow permit everything for Googlebot?

Google's robots.txt specification says an Allow rule without a path is ignored. You need a matching path to create an exception.

Can I use robots.txt to protect private content?

Google's robots.txt introduction describes crawler compliance as voluntary and recommends password protection for private files. You need an access control to secure them.

Sources

1. IETF, "Robots Exclusion Protocol", September 2022
2. Google, "How Google interprets the robots.txt specification", updated 31 August 2026
3. Google Search Central, "Introduction to robots.txt", updated 10 December 2025

---

Check how a page reads to answer engines. Open the GEO score analyzer