How quickly do independent Amazon crawler settings take effect?

Definition: Amazon documents three independently configurable user agents, Amazonbot, Amzn-SearchBot and Amzn-User. Its bot documentation says a change to one setting may take about 24 hours for Amazon's systems to reflect it, while the same documentation describes a cached host-level robots.txt copy from the last 30 days.

Key facts

• Amazon's bot documentation says the Amazonbot, Amzn-SearchBot and Amzn-User settings are independent, and changes may take about 24 hours to appear in its systems.
• The same page says the listed agents fetch a host-level robots.txt file or use a cached copy from the last 30 days.
• Amazon gives Amzn-SearchBot a specific fallback: when robots.txt does not mention it but allows other search bots, Amzn-SearchBot follows those other search-bot directives.
• Amazon describes Amzn-User as supporting user actions that may fetch live web information, and says it may not follow every robots.txt directive because those actions can start with a user.
• RFC 9309 requires case-insensitive matching of a crawler's product token to a robots.txt group and specifies the * group when no specific group matches.

What does an independent setting control?

An independent setting is a robots.txt group addressed to one product token. Amazon publishes separate example user-agent strings for Amazonbot, Amzn-SearchBot and Amzn-User, so a site can express separate rules for each token. RFC 9309 says crawlers match the product token to the relevant group, then obey that group's rules.

This pattern illustrates separate controls for the three Amazon agents:

User-agent: Amazonbot
Disallow: /

User-agent: Amzn-SearchBot
Allow: /

User-agent: Amzn-User
Disallow: /

The example blocks Amazonbot and Amzn-User while allowing Amzn-SearchBot. It is a configuration pattern, so the actual effect depends on the file served by the relevant host and on the agent that makes the request.

What is the propagation time?

Amazon describes the propagation time as approximately 24 hours for its systems to reflect a changed setting. That is the only per-setting timing figure on the bot page. It describes system reflection, rather than a scheduled request time for each host.

The same page also says these agents can use a cached copy of the host-level file from the last 30 days. RFC 9309 gives a general protocol recommendation that crawlers should not use a cached version for more than 24 hours unless the file is unreachable. The 30-day statement is Amazon's published crawler behavior; the RFC line is the general protocol guidance referenced by Amazon. Amazon does not publish an exact first-refetch schedule for each agent.

How should you check a change?

Serve the rule at the host-level path that Amazon names, such as example.com/robots.txt. Amazon says that a domain with multiple hosts exposes rules under each host, so site.example.com has its own site.example.com/robots.txt control point.

Check server logs for the relevant product token. Amazon's documented example strings include these forms:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amzn-SearchBot/0.1) Chrome/W.X.Y.Z Safari/537.36
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amzn-User/0.1) Chrome/W.X.Y.Z Safari/537.36

A missing explicit Amzn-SearchBot group has a special consequence in Amazon's documentation: if other search bots are allowed, Amzn-SearchBot uses those directives. Amazon also states that these user agents do not support the crawl-delay directive, so crawl-delay is not a propagation control for them.

Related terms

• Amazonbot
• Amzn-SearchBot
• Amzn-User
• How long can Amazon crawlers use a cached robots.txt file?

FAQ

Does changing Amazonbot also change Amzn-SearchBot?

No. Amazon describes each user-agent setting as independent. Amzn-SearchBot has its own group when named, plus the documented fallback to other search-bot rules when it is omitted.

Can crawl-delay control when an Amazon setting takes effect?

No. Amazon's bot page says the listed user agents do not support the crawl-delay directive. The documented propagation figure is about 24 hours for systems to reflect a setting change.

Does Amazon promise a request exactly 24 hours after an edit?

The page describes an approximate reflection time for its systems. It does not give a fixed request schedule for Amazonbot, Amzn-SearchBot or Amzn-User.

Sources

1. Amazon Developer, "About AmazonBot", undated
2. IETF, "RFC 9309: Robots Exclusion Protocol", September 2022

---

Check how a page reads to answer engines. Open the GEO score analyzer