robots.txt caching and error handling

Definition: robots.txt caching is a crawler's reuse of a previously fetched file containing crawl rules. Error handling determines whether a failed fetch leads the crawler to reuse those rules, suspend crawling, or proceed without crawl restrictions.

Key facts

• RFC 9309 recommends using a cached robots.txt for no more than 24 hours, with an exception when the file is unreachable.
• As of 17 October 2026, Google's robots.txt documentation describes a 12-hour crawling pause after a server error, followed by reuse of the last good version for the next 30 days.
• RFC 9111, HTTP Caching, defines the max-age response directive in seconds; it marks when a response becomes stale.

What does RFC 9309 require after a failed fetch?

The Robots Exclusion Protocol defines how crawlers interpret robots.txt. Its access results rules distinguish an unavailable file from an unreachable file. Unavailable means the server returns a response indicating the resource is unavailable, such as a client error. Unreachable means a server or network error prevents access; HTTP errors in the 500-599 range are an example. For an unreachable file, the RFC requires crawlers to assume complete disallow.

The RFC also permits caching. Its recommendation against using a copy older than 24 hours has an explicit exception for an unreachable file. After a reasonably long failure, with 30 days given as an example, a crawler may treat the file as unavailable or continue using a cached copy. That example leaves implementation choices to the crawler.

A parsing error has a different outcome: crawlers must try to parse each line and use the parseable rules. A malformed line therefore leaves other usable rules in effect.

How does Google handle server and network errors?

Google's error handling table gives an operational sequence for server errors:

1. For the first 12 hours, Google stops crawling the site while retrying robots.txt.
2. If fetching still fails, Google uses the last good version for the next 30 days and continues retrying. Without a cached version, it assumes there are no crawl restrictions during this stage.
3. If errors persist after 30 days, site availability determines the outcome. For a generally available site, Google acts as though robots.txt is absent and keeps checking for a new file. With general site availability problems, it stops crawling while periodically requesting robots.txt.

The same documentation treats DNS failures and networking problems, including timeouts and interrupted connections, as server errors. It also says a 503 response prompts fairly frequent retries.

What does caching change when rules are edited?

Google's caching documentation says a cached response may be shared by different crawlers. Google generally caches the file for up to 24 hours and may retain it longer when refreshing fails. The max-age directive in the Cache-Control HTTP response header may increase or decrease that lifetime.

The RFC's rule syntax supports this file, which asks crawlers covered by the wildcard group to avoid every path:

User-agent: *
Disallow: /

Because crawlers may reuse an earlier copy, publishing these lines does not immediately replace cached rules. After a successful download, the RFC requires the crawler to follow the parseable rules in the fetched file.

Related terms

• robots.txt
• HTTP status codes
• Googlebot
• What happens when robots.txt returns a server error?

FAQ

Can you use a 403 response to block Google's crawling through robots.txt?

Google's client error documentation says 4xx responses other than 429 imply no crawl restrictions. Returning 403 for robots.txt therefore causes Google to treat the file as absent.

Is 24 hours a fixed expiry for Google's cached rules?

The same Google documentation allows longer retention when refreshing fails. It also allows cache lifetime adjustments based on max-age response headers.

Sources

1. IETF, "RFC 9309: Robots Exclusion Protocol", September 2022
2. Google, "How Google interprets the robots.txt specification", updated 31 August 2026
3. IETF, "RFC 9111: HTTP Caching", June 2022

---

Check how a page reads to answer engines. Open the GEO score analyzer