In short. When robots.txt omits Amzn-SearchBot but allows other search bots, Amazon says it follows the directives given to those other search bots.
Definition: Amazon documents a fallback for Amzn-SearchBot: when robots.txt omits its token but permits other search bots, Amzn-SearchBot crawls according to the directives given to those other search bots. The Robots Exclusion Protocol supplies the general wildcard and path-matching rules used when a crawler has no matching group.
Key facts
- Amazon says Amzn-SearchBot is used to improve search experiences in Amazon products and services, including experiences such as Alexa. Amazon’s bot documentation gives the fallback rule for a file that names other search bots but omits Amzn-SearchBot.
- The Robots Exclusion Protocol requires case-insensitive matching for a crawler’s product token and says matching groups are combined.
- RFC 9309 assigns the
User-agent: *group when no group matches the crawler token and a wildcard group exists. If neither a matching group nor a wildcard group exists, it says no rules apply. - Amazon reads robots.txt at host level. A domain with multiple hosts can expose separate rules at addresses such as
example.com/robots.txtandsite.example.com/robots.txt. - Amazon says its listed automated agents use a host-level file or a cached copy from the last 30 days, and treat an unfetchable file as absent.
How does the fallback work?
Amazon’s rule applies when the file omits Amzn-SearchBot and allows other search bots. In that situation, Amazon says the crawler follows the robots.txt directives given to those other search bots. The page leaves priority among several named search-bot groups unspecified, so an explicit Amzn-SearchBot group is the clearest way to state a separate policy.
The standard defines the ordinary group-selection sequence. A crawler first looks for a case-insensitive product-token match. If it finds no matching group, RFC 9309 directs it to the wildcard group when one is present. A file can therefore contain a general policy and a more specific policy for a named bot:
User-agent: *
Disallow: /private/
User-agent: Googlebot
Allow: /private/public/
With Amzn-SearchBot absent, Amazon’s documentation is the source for its search-bot fallback. RFC 9309 is the source for the wildcard rule when no named group applies.
Which paths are allowed inside the selected group?
After group selection, RFC 9309 says the crawler matches the path from its first character and uses the most specific matching rule. A longer path pattern takes precedence over a shorter one. If an allow and disallow rule are equivalent, the standard says the allow rule should be used. If no rule matches the requested URI, the URI is allowed.
This path evaluation happens after deciding which group applies. A wildcard group can control Amzn-SearchBot under the standard, while Amazon’s separate fallback statement covers a file that names other search bots. A Disallow line in an unrelated named group has no universal effect solely because it exists.
Which robots.txt file does Amazon read?
Amazon’s documentation says these listed automated agents use host-level robots.txt files. Google’s robots.txt specification likewise states that rules apply only to the host, protocol, and port where the file is hosted. A rule at https://example.com/robots.txt therefore has a different scope from one at https://www.example.com/robots.txt.
Amazon also documents caching and retrieval behavior: its agents can use a cached copy from the last 30 days, and on failed retrieval they behave as if the file were absent. Google documents its own handling separately: it treats most 4xx responses, except 429, as an absent robots.txt file. Those Google response rules describe Googlebot, while Amazon’s page supplies the behavior for Amazon’s listed agents.
How can a site state the policy explicitly?
Add a group for the exact product token when the intended policy should be independent of other search bots:
User-agent: Amzn-SearchBot
Disallow: /private/
The token is the product name Amazon publishes in its example user-agent string, where Amzn-SearchBot/0.1 appears in a browser-compatible identification string. Amazon says each user-agent setting is independent of the others, and changes may take about 24 hours for its systems to reflect.
Related terms
FAQ
If robots.txt names Googlebot but omits Amzn-SearchBot, what does Amazon use?
Amazon says Amzn-SearchBot crawls in accordance with the directives given to other search bots when its own token is absent and other search bots are allowed. The page leaves priority unspecified when several named search-bot groups appear.
Does User-agent: * apply to Amzn-SearchBot?
Under RFC 9309, a crawler uses the wildcard group when no matching product-token group exists and a wildcard group is present. Amazon separately documents its fallback to directives given to other search bots.
Does this rule also describe Amzn-User?
No single fallback statement should be transferred to Amzn-User. Amazon documents Amzn-User separately and says user-initiated actions may cause it to not follow all robots.txt directives.
Sources
- Amazon, "About AmazonBot", undated
- Internet Engineering Task Force, "Robots Exclusion Protocol", September 2022
- Google Search Central, "How Google interprets the robots.txt specification", 31 August 2026
Check how a page reads to answer engines. Open the GEO score analyzer