In short. Amzn-User may depart from robots.txt directives for user-initiated actions, so a disallow rule alone cannot enforce restricted access.
Definition: Amzn-User is Amazon's web fetcher for user actions, including Alexa queries that need current information. As of 17 October 2026, Amazon states that these user-initiated actions may not follow all robots.txt directives.
Key facts
- Amazon's About AmazonBot documentation explicitly qualifies Amzn-User's robots.txt compliance because users can initiate its actions.
- The Robots Exclusion Protocol standard defines robots.txt rules as instructions for crawlers and states that they are not access authorization.
- Amazon publishes a live-crawl IP address list, linked from its Amzn-User documentation, with entries under
prefixes.
What is the user-initiated exception?
Amazon's Amzn-User description gives an Alexa question requiring up-to-date information as an example: the agent may fetch live web information on the user's behalf. Amazon connects the robots.txt exception directly to this user-initiated activity.
The same page says automated crawling by the listed agents respects the Robots Exclusion Protocol, the system for publishing crawler instructions in robots.txt. The Amzn-User qualification therefore matters when interpreting that general statement. Amazon does not enumerate the directives bypassed or the exact conditions for bypassing them.
What does an Amzn-User rule express?
A user-agent token is the crawler name used to select a group of robots.txt rules. RFC 9309, section 2.2.1, defines this matching mechanism. A group addressing Amzn-User can express a request to disallow access to every path:
User-agent: Amzn-User
Disallow: /
This configuration combines the standard's group syntax with Amazon's published agent name. Its effect remains subject to the user-initiated exception.
Amazon's documentation says each agent's setting is independent. A rule for Amazonbot therefore needs separate consideration from a rule for Amzn-User. The same documentation says setting changes may take approximately 24 hours to appear in Amazon's systems and that the listed agents do not support crawl-delay.
How does a website enforce restricted access?
RFC 9309, section 3, directs sites to use security measures appropriate to the application layer for access control, giving HTTP authentication as an example. It also warns that listing paths in robots.txt exposes those paths publicly. Content requiring restricted access belongs behind an enforced access control mechanism.
A robots.txt entry communicates crawling policy. Authentication determines whether a request can obtain protected content. That distinction applies when interpreting an Amzn-User disallow rule alongside Amazon's stated exception.
Related terms
FAQ
Can I rely on Disallow to stop every Amzn-User request?
No. Amazon explicitly allows for Amzn-User actions that depart from robots.txt directives because a user can initiate them.
Should I put confidential paths in robots.txt?
RFC 9309 warns that these paths become public and identifies HTTP authentication as an access control measure. Use enforced access controls for confidential content.
Sources
- Amazon, "About AmazonBot", undated
- IETF, "RFC 9309: Robots Exclusion Protocol", September 2022
- Amazon, "Amazonbot Live Crawl IP addresses", undated
Check how a page reads to answer engines. Open the GEO score analyzer