In short. Amazon documents a robots.txt cache window covering the last 30 days for its listed automated user agents.
Definition: Amazon documents a cached robots.txt window of the last 30 days for its listed automated user agents. The file is read at host level, and Amazon states that its user agents behave as if the file does not exist when it cannot be fetched.
Key facts
- Amazon's Amazonbot documentation says its listed automated user agents can use a cached host-level robots.txt copy from the last 30 days.
- The same documentation says those user agents respect the Robots Exclusion Protocol and its user-agent, allow and disallow directives.
- Amazon says each user-agent setting is independent and changes may take about 24 hours for its systems to reflect.
- RFC 9309 says crawlers should not use a cached copy for more than 24 hours unless robots.txt is unreachable.
- Google documents a separate policy: it generally caches robots.txt for up to 24 hours and may keep it longer when a refresh is not possible.
What is Amazon's documented cache window?
Amazon's page for Amazonbot, Amzn-SearchBot and Amzn-User says automated crawling by these listed user agents respects the Robots Exclusion Protocol. It then gives the operational rule: the agents fetch host-level robots.txt files or use a cached copy from the last 30 days. For Amazon's own documented behavior, 30 days is the relevant cache window.
The page also says that when a file cannot be fetched, these user agents behave as if it does not exist. Amazon does not describe a more detailed refresh schedule within that window. It separately says that each user-agent setting is independent and that a setting change may take about 24 hours for its systems to reflect. That propagation statement concerns changes to Amazon's controls, while the 30-day statement concerns a cached robots.txt file.
How does this relate to the robots.txt standard?
RFC 9309, the September 2022 Robots Exclusion Protocol, permits crawlers to cache fetched robots.txt contents. Its caching rule says crawlers should not use a cached version for more than 24 hours unless the file is unreachable. The exception matters because the same RFC says a robots.txt file unreachable through server or network errors means complete disallow, with HTTP 500 to 599 responses as an example.
The RFC also gives a longer-failure example. If robots.txt remains undefined for a reasonably long period, such as 30 days, a crawler may treat it as unavailable or continue using a cached copy. This is protocol guidance for crawlers in general. Amazon's page supplies the service-specific statement for its listed agents, so the Amazon documentation and the RFC should be read at their respective scopes.
Which host and rules does Amazon read?
Amazon says its user agents look for robots.txt at the host level. For example.com, that means:
https://example.com/robots.txt
Rules exposed under another host are handled separately. The page gives site.example.com/robots.txt as the location for a separate site.example.com host. Amazon also says automated crawling honors the user-agent, allow and disallow directives. A rule scoped to Amazonbot can therefore use the standard group syntax:
User-agent: Amazonbot
Disallow: /private/
RFC 9309 requires a crawler to follow parseable rules after a successful download. A cached copy can therefore continue to represent the earlier rules until Amazon obtains a newer file or applies the failure behavior described on its page.
What can you compare with Google's behavior?
Google's robots.txt specification says Google generally caches the file for up to 24 hours. It may cache it longer when refreshing is impossible because of timeouts or 5xx errors, and it may adjust the lifetime using the max-age value in the HTTP Cache-Control header. Google also says that a cached response may be shared by different crawlers.
Google's published policy shows why a crawler's own documentation matters. For server errors, it stops crawling during the first 12 hours while retrying, then uses the last good version for the next 30 days while continuing to fetch. Amazon publishes a different cache statement for its own agents.
Related terms
FAQ
Is Amazon's 30-day window the same as the RFC 9309 cache recommendation?
No. RFC 9309 gives a general 24-hour recommendation with an exception for an unreachable file. Amazon's documentation states that its listed automated user agents can use a cached copy from the last 30 days.
Does a robots.txt change reach Amazon crawlers immediately?
Amazon says changes to each user-agent setting may take about 24 hours for its systems to reflect. Its page separately documents a cached robots.txt copy from the last 30 days.
Does Amazon use one robots.txt file for every host?
No. Amazon says its user agents read robots.txt at the host level, giving example.com/robots.txt and site.example.com/robots.txt as separate examples.
Sources
- Amazon Developer, "About AmazonBot", undated
- IETF, "RFC 9309: Robots Exclusion Protocol", September 2022
- Google for Developers, "How Google interprets the robots.txt Specification", 31 August 2026
Check how a page reads to answer engines. Open the GEO score analyzer