What happens when robots.txt returns a server error?

Definition: A robots.txt server error prevents a crawler from retrieving the file that controls crawl access. Google initially pauses crawling; subsequent handling depends on cached rules and site availability.

Key facts

• As of 17 October 2026, Google's robots.txt specification describes an initial 12-hour crawl pause for server errors.
• The IETF's RFC 9309 requires crawlers to assume complete disallow when robots.txt is unreachable through server or network errors.
• Amazon's crawler documentation, reviewed in October 2026, says automated crawlers can use a cached file from the last 30 days and treat a file they cannot fetch as absent.
• Google's robots.txt report documentation describes fetch history and a request to recrawl after a fetch error is fixed.

How does Google handle a prolonged failure?

Google's specification gives this sequence for HTTP 5xx server errors:

1. During the first 12 hours, crawling stops while robots.txt fetch attempts continue.
2. For the next 30 days, Google uses the last good version while retrying. Without a cached version, it assumes no crawl restrictions.
3. After 30 days of errors, a generally available site is treated as having no robots.txt. General site availability problems instead cause crawling to stop. File checks continue in both cases.

A 503 response prompts frequent retries. Google also treats DNS failures and network timeouts as server errors.

What does the protocol require?

The Robots Exclusion Protocol is the standard for communicating crawler access rules. RFC 9309, section 2.3.1.4, classifies HTTP responses in the 500-599 range as server errors and requires complete disallow when the file is unreachable.

After a reasonably long failure, with 30 days given as an example, the standard permits treating the file as unavailable or continuing with a cached copy. Section 2.4 permits keeping a cached version beyond 24 hours when the file is unreachable.

Section 2.3 places the lowercase file at the service root, with this example URL:

https://www.example.com/robots.txt

How does Amazon describe fetch failures?

Amazon's About AmazonBot page covers Amazonbot and Amzn-SearchBot, its search crawler. Its policy for automated crawling allows fetching the host's robots.txt or using a copy cached within the last 30 days. It says a file that cannot be fetched is treated as nonexistent.

That wording describes general fetch failure; Amazon provides no separate schedule for individual server-error codes. The same page locates rules at each host: example.com and site.example.com each have their own robots.txt. Operational checks therefore need to cover the host the crawler requests.

How can a website check recovery?

Google's robots.txt report shows the file URL, latest fetch status and the time of the latest attempt. Clicking a file opens its last fetched contents; selecting Versions shows fetch requests from the last 30 days. The history includes a request only when the fetched file or result changes.

The report distinguishes Fetch status, which concerns retrieval, from Issues, which concerns parsing the retrieved rules. This helps separate an inaccessible file from a file containing invalid lines.

After a fetch error is fixed, the file's menu offers Request a recrawl. The documented procedure is to select the more settings icon beside the file, then select that action.

Related terms

• robots.txt
• robots.txt caching and error handling
• HTTP status codes
• Amazonbot

FAQ

Does a missing file have the same effect as a server error?

Google's specification treats HTTP 4xx responses, except 429, as an absent file with no crawl restrictions.

Can I inspect the version Google fetched before the failure?

Yes. Google's report documentation describes opening the last fetched file and selecting Versions to inspect earlier fetch results.

Sources

1. Google, "How Google interprets the robots.txt specification", updated 31 August 2026
2. IETF, "RFC 9309: Robots Exclusion Protocol", September 2022
3. Amazon, "About AmazonBot", accessed 17 October 2026
4. Google, "robots.txt report", accessed 17 October 2026

---

Check how a page reads to answer engines. Open the GEO score analyzer