In short. robots.txt publishes crawler access rules through User-agent, Allow and Disallow lines at a website’s top-level path.

Definition: robots.txt is a text file that publishes rules about which URL paths automated clients, called crawlers, may access on a website. It uses the Robots Exclusion Protocol, specified in RFC 9309.

Key facts

The protocol specification requires a lowercase /robots.txt file at the service's top-level path, encoded in UTF-8 and served as text/plain.
A User-agent line identifies the crawler a group of rules addresses; Allow and Disallow specify permitted and restricted paths, according to RFC 9309.
As of 17 October 2026, Google's guidance states that a blocked web page's URL can still appear in search results when other pages link to it.
The protocol's security guidance recommends measures such as HTTP authentication to protect access to content; paths listed in robots.txt are public.

How do the rules work?

A group contains crawler names followed by access rules. In RFC 9309, * in a User-agent line supplies the fallback group when no crawler-specific group matches. Path matching uses the most specific matching rule, measured by its length in bytes. For equivalent Allow and Disallow rules, the specification recommends using Allow.

This excerpt matches the specification's example for the fictional crawler foobot:

User-Agent: foobot
Disallow:/
Allow:/example/page.html
Allow:/example/allowed.gif

The example restricts the crawler to URL paths beginning with /example/page.html or /example/allowed.gif. These are path prefixes: a rule without an end-of-match marker can also match a longer path. The same specification defines $ as the end-of-match marker.

How does robots.txt affect AI search?

As of 17 October 2026, OpenAI's crawler documentation identifies OAI-SearchBot as its search crawler and GPTBot as a crawler for content that may be used in model training. The settings are independent: a site can allow search crawling while disallowing GPTBot.

OpenAI says sites opting out of OAI-SearchBot are excluded from ChatGPT search answers, although they can still appear as navigational links. Its documentation separately describes ChatGPT-User requests as user-initiated actions for which robots.txt rules may not apply. Choosing a crawler name therefore requires checking which activity that name controls.

Does blocking crawling remove a page from search?

Google's robots.txt introduction explains that links from other pages can let Google index a blocked URL without visiting its content. The search result can appear without a description.

For exclusion from Google Search, the guide points to noindex, an instruction to exclude content from indexing, or password protection. For private content, authentication controls access at the server. The protocol specification explicitly states that robots.txt rules are not access authorization.

How is the file checked?

Google's file creation guide recommends opening the published file in a private browser window to confirm public access. It also identifies the robots.txt report in Search Console for files already accessible on a site, and Google's open source robots.txt library for local testing.

The same guide limits a file's scope to its protocol, host and port. A file at https://example.com/robots.txt applies to that HTTPS host; separate subdomains and HTTP addresses require their own applicable rules.

FAQ

Can you protect confidential files with robots.txt?

Use authentication for confidential files. RFC 9309 explains that listing paths in robots.txt exposes those paths publicly and recommends application-level security measures.

Do you submit a new robots.txt file to Google?

After you upload and test the file, Google's crawlers automatically find and use it, according to the creation guide. The guide also links to instructions for refreshing Google's cached copy after an update.

Sources

Check how a page reads to answer engines. Open the GEO score analyzer