In short. Amazon says its listed automated user agents behave as if robots.txt does not exist when they cannot fetch it.
Definition: Amazon documents that its listed automated user agents use a host-level
robots.txtfile or a cached copy from the last 30 days. When the fetch fails, they treat the file as absent.
Key facts
- Amazon's documentation covers Amazonbot, Amzn-SearchBot and Amzn-User, and says each user agent setting is independent and may take about 24 hours to reach Amazon's systems. Amazon's bot documentation gives the scope.
- Amazon says automated crawling from these listed user agents respects the Robots Exclusion Protocol, uses host-level rules or a cached copy from the last 30 days, and treats an unfetchable file as absent. The Robots Exclusion Protocol defines the standard that Amazon names.
- RFC 9309 describes an unavailable file in the 400 to 499 range as allowing a crawler to access any server resource, while a server or network error makes the file undefined and requires complete disallow. Google's documentation describes its own cache and error handling separately. Google's robots.txt documentation records that implementation's behavior.
What does Amazon document?
Amazon's page describes controls for Amazonbot, Amzn-SearchBot and Amzn-User. It says that automated crawling from these listed user agents honors the user-agent, allow and disallow directives. A host-level file belongs to the host being accessed, so example.com/robots.txt and site.example.com/robots.txt are separate locations when both hosts exist. Amazon also says that Amzn-User can be initiated by a user and may bypass some robots.txt directives.
The page gives this example user agent string for Amzn-SearchBot:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amzn-SearchBot/0.1) Chrome/W.X.Y.Z Safari/537.36
What follows a failed fetch?
Amazon says its listed user agents fetch a host-level robots.txt file or use a cached copy from the last 30 days. When a fetch fails, the documented behavior is to act as if the file is absent. In practical terms, the crawler has no fetched Amazon allow or disallow rules from that file to apply at that time. Amazon's page leaves the relevant HTTP response codes and network errors unspecified.
RFC 9309 separates two cases. A 400 to 499 response is an “unavailable” status, for which a crawler may access any resources on the server. A server or network error, including the 500 to 599 range, makes the file undefined and requires complete disallow. Amazon names this RFC as the protocol it respects, while separately publishing its own missing-file outcome.
How does Google's documented handling compare?
Google says that after a failed fetch of a new version, it uses the last good version for the next 30 days while continuing to fetch. If no cached version is available, Google assumes there are no crawl restrictions. After 30 days, Google behaves as if there is no robots.txt file when the site is generally available, but stops crawling when the site has general availability problems. Its documentation says Google generally caches robots.txt for up to 24 hours and may cache it longer when refreshing is impossible.
These Google rules show why a website's observed result needs Amazon-specific evidence rather than a generic robots.txt rule alone. The relevant Amazon user agent, host, response and cache state must be recorded together.
What should a website operator check?
Check the robots.txt response separately for every host that receives requests. Record whether the response was fetched, whether the requested host returned the file, and whether the request used an Amazon user agent. For Amzn-SearchBot, Amazon says that when the file omits that token but allows other search bots, Amzn-SearchBot follows the directives given to those other bots.
For a temporary outage, the important documented Amazon result is the missing-file behavior. A site owner can compare the server log with the file served at the host-level path and keep the test scoped to the named Amazon user agent. The page also says that each user agent setting is independent and that changes may take about 24 hours to reach Amazon's systems.
Related terms
FAQ
Does an unavailable robots.txt file automatically mean complete disallow for Amazon?
Amazon's published rule says that its listed automated user agents treat an unfetchable file as absent. RFC 9309 has a different complete-disallow rule for an unreachable file caused by server or network errors, so the Amazon page's wording should be kept separate from the RFC's status categories.
Does Amazon use one robots.txt file for every subdomain?
No. Amazon documents host-level fetching and gives example.com/robots.txt and site.example.com/robots.txt as separate examples for separate hosts.
Does this rule cover Amzn-User requests made for a person?
Amazon says Amzn-User supports user actions such as Alexa queries that need current information, and that it may bypass some robots.txt directives because those actions can be initiated by a user.
Sources
- Amazon Developer, "About AmazonBot", undated
- Internet Engineering Task Force, "Robots Exclusion Protocol", September 2022
- Google for Developers, "How Google interprets the robots.txt specification", 31 August 2026
Check how a page reads to answer engines. Open the GEO score analyzer