How does Applebot react when a website slows down or returns errors?

Definition: Applebot automatically adjusts its crawl rate when a website slows down or returns errors. Apple describes this as a way to reduce the impact of crawling on site owners, while leaving the rate, threshold and recovery interval undocumented.

Key facts

• Apple's About Applebot documentation says that Applebot's crawl rate adjusts automatically when a site slows down or returns errors.
• Apple says that Applebot caches crawled content to reduce unnecessary crawling.
• Applebot does not follow the crawl-delay directive in robots.txt, according to the same documentation.
• RFC 6585 defines HTTP 429 Too Many Requests as rate limiting after too many requests in a given amount of time.
• RFC 9110 defines HTTP 503 Service Unavailable for temporary overload or scheduled maintenance and permits a Retry-After header.

What does Apple document about automatic adjustment?

Apple says Applebot is engineered to minimize its impact on site owners. Its example is explicit: when a site slows down or returns errors, Applebot's crawl rate adjusts automatically. The same paragraph says Applebot caches crawled content to reduce unnecessary crawling. The documentation gives no numeric crawl rate, threshold, or interval for the adjustment.

The statement describes a crawler response to observed site conditions. It does not define a fixed delay for a particular error code. A slow response and an error response are both conditions Apple names, so server logs should retain response time and status code alongside the Applebot request.

Which HTTP responses describe overload or rate limiting?

HTTP standards give names to two server conditions that can appear in the errors covered by Apple's description. RFC 6585 defines 429 Too Many Requests as too many requests in a given amount of time. Its response can include a Retry-After header indicating how long to wait before another request.

HTTP/1.1 429 Too Many Requests
Content-Type: text/html
Retry-After: 3600

RFC 9110 defines 503 Service Unavailable as a server temporarily unable to handle a request because of overload or scheduled maintenance. A server may send Retry-After to suggest how long a client should wait before retrying. These standards define the response meanings; Apple's documentation does not state how Applebot interprets a Retry-After value.

Does robots.txt set Applebot's crawl rate?

Apple says that Applebot respects standard robots.txt directives targeted at Applebot, and its example excludes two URL paths:

User-agent: Applebot
Allow: /
Disallow: /private/
User-agent: *
Disallow: /not-allowed/

The same Apple documentation states that Applebot does not follow crawl-delay. A robots group can therefore control which paths Applebot requests, while the documented response to a slow or erroring site remains automatic rate adjustment. If Applebot-specific instructions are absent, Apple says it follows Googlebot instructions when those are present.

How can a site owner inspect the condition?

Apple explains that a user agent helps webmasters identify crawler traffic and obtain accurate access log reports. An investigation can therefore group verified Applebot requests by response status and response time, then compare those observations with the site's periods of overload or maintenance. The HTTP standards define 429 and 503 meanings, but Apple publishes no mapping from either status to a specific Applebot crawl interval.

Related terms

• Applebot
• Applebot-Extended
• Crawl-delay
• How should a server respond when a crawler sends too many requests?

FAQ

Does Applebot follow Crawl-delay?

Apple's documentation says that Applebot does not follow crawl-delay. URL access rules in robots.txt remain available for paths that should not be crawled.

Does a 429 or 503 tell Applebot exactly how long to wait?

If you return 429 or 503, the HTTP standards describe the condition and allow a Retry-After value. Apple has not documented how Applebot applies that value, so the entry does not assign a wait time to either status.

Sources

1. Apple, "About Applebot", 4 September 2026
2. IETF, "Additional HTTP Status Codes", April 2012
3. IETF, "HTTP Semantics", June 2022

---

Check how a page reads to answer engines. Open the GEO score analyzer