How do I slow down Anthropic's crawling?

Definition: As of 17 October 2026, Anthropic documents support for Crawl-delay, a nonstandard extension placed in robots.txt, the file that gives crawlers instructions for a website. Its configuration example applies the delay to ClaudeBot.

Key facts

• Anthropic's crawler privacy documentation gives the example User-agent: ClaudeBot followed by Crawl-delay: 1.
• The Robots Exclusion Protocol, RFC 9309, requires the lowercase filename /robots.txt at the service's top-level path.
• As of 17 October 2026, Google's robots.txt specification explicitly lists crawl-delay among unsupported fields.

What configuration does Anthropic show?

Anthropic's privacy documentation provides this example for limiting crawling activity:

User-agent: ClaudeBot
Crawl-delay: 1

The User-agent line identifies the crawler to which the instruction applies. Anthropic describes respecting Crawl-delay "where appropriate". Its page supplies the value 1 without specifying units, an accepted range, or a rule-refresh interval.

The same documentation uses Disallow: / to block a bot from the entire website. Adding that exclusion changes the configuration from a request to limit activity to a request to block crawling.

Where does the file belong?

RFC 9309 requires a UTF-8 encoded file served with the media type text/plain. Its example location is:

https://www.example.com/robots.txt

The filename is lowercase, and the file sits directly beneath the host's root. These requirements provide concrete deployment checks: the public URL, filename, encoding and response media type should match the protocol.

The protocol also permits crawlers to interpret additional records beyond its defined rules. Anthropic identifies Crawl-delay as such a nonstandard extension; support therefore depends on the crawler's implementation.

How should existing crawler rules be handled?

A robots.txt group associates a crawler name with rules. RFC 9309 calls the crawler name a product token and requires case-insensitive matching. Its fallback group uses User-agent: * when no named group matches.

When adding a ClaudeBot group, review existing path restrictions alongside the delay instruction. Under the protocol's group-selection rules, a named matching group takes precedence over the wildcard fallback. The protocol also combines rules from multiple groups matching the same crawler name. These details matter when an existing file already contains a ClaudeBot group or wildcard exclusions.

Related terms

• Crawl-delay
• ClaudeBot
• robots.txt
• Where does a robots.txt file need to be hosted?

FAQ

Can I use the same delay rule for Googlebot?

Google's robots.txt specification says crawl-delay is unsupported. You need to check the documented support of each crawler before using this field for it.

Can I block ClaudeBot completely instead?

Yes. Anthropic's documented blocking example pairs User-agent: ClaudeBot with Disallow: /. For a whole-site block, Anthropic instructs you to apply the configuration separately to each subdomain concerned.

Sources

1. Anthropic, "Does Anthropic crawl data from the web, and how can site owners block the crawler?", 7 April 2026
2. IETF, "Robots Exclusion Protocol", September 2022
3. Google, "How Google interprets the robots.txt specification", 31 August 2026

---

Check how a page reads to answer engines. Open the GEO score analyzer