In short. A robots.txt Disallow rule excludes matching paths from crawling, subject to group selection and the most specific applicable Allow or Disallow rule.

Definition: The robots.txt Disallow directive is a crawl instruction that excludes matching URL paths for the crawlers addressed by its group. The Robots Exclusion Protocol, specified in RFC 9309, defines how crawlers select and compare these rules.

Key facts

RFC 9309 requires matching from the beginning of the path and selection of the matching rule with the most octets, or bytes.
As of 17 October 2026, Google's robots.txt specification states that a Disallow rule with an empty path is ignored.
Google's robots.txt introduction explains that a blocked page's URL can still appear in search results when linked from elsewhere.

How is a Disallow rule written?

A group associates rules with crawlers through a User-agent line. RFC 9309 requires the file at /robots.txt in the service's top-level path, encoded as UTF-8 and served as text/plain. Its example includes this group:

User-Agent: foobot
Disallow:/
Allow:/example/page.html
Allow:/example/allowed.gif

Here, Disallow:/ excludes paths starting with /. The Allow rules permit the longer matching prefixes shown. The standard also implicitly allows /robots.txt itself. Spaces after the colon are optional. These example paths remain prefixes: a filename in a rule does not automatically make the match exact.

The same standard requires crawlers to combine groups matching their product token, the crawler name used to select rules. A User-agent: * group applies when no specific group matches. Rules placed before the first User-agent line should be ignored.

How do path exclusions match URLs?

Google's matching examples show that /fish matches /fish, /fish.html and /fishheads. Adding a trailing slash changes the prefix: /fish/ matches /fish/salmon.htm but leaves the bare /fish unmatched. Google's comparison is case-sensitive, so /fish leaves /Fish.asp unmatched too.

A wildcard is a character that stands for other characters. * matches zero or more characters; $ anchors the match at the end of the URL. The same documentation shows that /*.php matches /folder/filename.php?parameters, while /*.php$ leaves that URL unmatched because the query follows the suffix. A trailing wildcard adds nothing to an ordinary prefix: /fish* is equivalent to /fish.

RFC 9309 recommends case-sensitive path comparison and favors Allow when equivalent Allow and Disallow rules conflict. It also requires encoding normalization before comparison: percent-encoded unreserved ASCII characters are decoded, while reserved characters retain their encoding. Its example compares /foo/bar/%62%61%7A as /foo/bar/baz.

What does the exclusion control?

Google's robots.txt introduction identifies crawl traffic management as the file's main purpose. Blocking scripts or styles can also affect Google's understanding of pages that depend on those resources.

For a web page linked from elsewhere, Google can index the URL without fetching its content. The documentation recommends an indexing exclusion such as noindex, or password protection, when the aim is to keep a page out of search results. Private information needs access controls: robots.txt instructions depend on crawler cooperation.

FAQ

Does an empty Disallow block the whole site?

No. Google's specification says crawlers ignore a rule without a path. Disallow: / supplies a path matching the root and lower-level URLs, subject to applicable Allow rules.

Can a longer Allow override Disallow?

Yes. Under RFC 9309, the most specific matching rule applies. Its foobot example permits /example/page.html despite Disallow:/.

Sources

Check how a page reads to answer engines. Open the GEO score analyzer