In short. Use a robots.txt group with Disallow: / for Amazonbot and a separate group with Allow: / for Amzn-SearchBot, reviewing existing rules for both.
Definition: As of 17 October 2026, Amazon documents independent access settings for Amazonbot, a crawler whose collection may support Amazon AI model training, and Amzn-SearchBot, its search crawler. Separate groups in robots.txt can disallow Amazonbot while explicitly allowing Amzn-SearchBot.
Key facts
- Amazon's About Amazonbot documentation states that each user agent setting is independent and that changes may take approximately 24 hours to reach its systems.
- The IETF's Robots Exclusion Protocol specification requires crawlers to combine rules from groups matching the same user agent.
- Amazon's Searchbot IP address page publishes network prefixes in
ip_prefixfields, providing a reference for checking the source addresses of search crawler requests.
Which robots.txt rules express this choice?
A robots.txt file contains instructions for crawlers, grouped by the crawler name in a User-agent line. A group consists of that line and the rules that follow it. The IETF's RFC 9309 defines Disallow and Allow as rules governing access to matching paths.
The following configuration applies those rules to Amazon's documented crawler names:
User-agent: Amazonbot
Disallow: /
User-agent: Amzn-SearchBot
Allow: /
The Amazonbot group disallows crawling across the site's paths. The Amzn-SearchBot group permits crawling across those paths. This example allows the search crawler sitewide; a site with path restrictions needs those restrictions in its Amzn-SearchBot group as well.
Existing groups with either name also need review. Under the same specification, matching groups are combined, and the most specific matching path rule applies. Adding a new group leaves existing rules relevant to the final result.
Why name Amzn-SearchBot explicitly?
Amazon's crawler documentation says Amazonbot improves its products and services and may be used to train Amazon AI models. It describes Amzn-SearchBot as supporting search experiences in Amazon products and services, including Alexa, and says that crawler does not collect content for AI model training.
Amazon also documents a fallback: when robots.txt omits Amzn-SearchBot but allows other search bots, Amzn-SearchBot follows the directives given to other search bots. An explicit group expresses the intended search access directly.
Amazon describes permitted content as eligible to appear in those search experiences. The configuration therefore expresses separate collection permissions; eligibility is the documented search outcome.
Where should the file be served, and when does it apply?
The IETF's access-method specification places the file at the service's top-level /robots.txt path, encoded as UTF-8 and served as text/plain.
Amazon's robots.txt guidance checks rules per host. Its examples distinguish example.com/robots.txt from site.example.com/robots.txt. A domain with multiple hosts therefore needs the intended rules on each relevant host.
The same guidance allows a cached copy from the last 30 days and says that, when the file cannot be fetched, the agents behave as though it is absent. Checking that the intended file is publicly retrievable is part of checking this configuration. The approximately 24-hour setting update period and the documented cache window are separate statements in Amazon's guidance.
Related terms
FAQ
Does this also block live retrieval for an Alexa question?
Amazon's Amzn-User documentation describes a separate agent that can fetch current information for an Alexa query. Because a user can initiate these actions, Amazon says Amzn-User may not follow all robots.txt directives.
Can I use crawl-delay instead of Disallow?
Amazon states in the same documentation that its listed agents do not support crawl-delay. The blocking instruction in this configuration is Disallow: / in the Amazonbot group.
Should I repeat restrictions from my wildcard group?
Review them when creating an explicit Amzn-SearchBot group. Under RFC 9309's group selection rules, the User-agent: * group applies when no matching named group exists, so restrictions needed for a named crawler belong in its applicable group.
Sources
- Amazon, "About Amazonbot", undated, accessed 17 October 2026
- IETF, "RFC 9309: Robots Exclusion Protocol", September 2022
- Amazon, "Amazon Searchbot IP addresses", undated, accessed 17 October 2026
Check how a page reads to answer engines. Open the GEO score analyzer