In short. Crawl-delay is a nonstandard robots.txt extension whose support and timing semantics depend on the crawler operator.

Definition: Crawl-delay is a nonstandard extension to robots.txt, the file through which websites give instructions to automated web clients called crawlers. It requests a limit on crawling activity, with support and timing semantics determined by the crawler operator.

Key facts

The support statements below reflect documentation available on 17 October 2026.

Anthropic's crawler documentation explicitly supports the nonstandard extension and provides a ClaudeBot example.
The Robots Exclusion Protocol standard, RFC 9309, permits crawlers to interpret additional records beyond the protocol's defined rules.
Google's robots.txt specification lists crawl-delay among unsupported fields.
Bing's Webmaster Blog explanation, published on 3 May 2012, documents support and describes a time window with at most one page crawled.

How does Anthropic document Crawl-delay?

Anthropic's crawler guidance presents the following example for limiting activity:

User-agent: ClaudeBot
Crawl-delay: 1

The User-agent line names ClaudeBot as the crawler addressed by this example. Anthropic says it respects Crawl-delay where appropriate. The article leaves the value's units, accepted range and precise scheduling semantics unspecified.

The same guidance provides a separate example for blocking ClaudeBot across a website:

User-agent: ClaudeBot
Disallow: /

It places the robots.txt file in the website's top-level directory and asks owners to apply blocking separately to each subdomain concerned. The two examples express different instructions: limiting crawling activity and excluding crawling across the site.

Why does support vary between crawlers?

The Robots Exclusion Protocol defines User-agent, Allow and Disallow. Its section 2.2.4 allows crawlers to interpret other records, provided parsing those records preserves the parsing of the standard records. Crawl-delay therefore depends on an operator's implementation of an extension.

Google's specification lists the supported fields as user-agent, allow, disallow and sitemap, and explicitly excludes crawl-delay. A Crawl-delay line addressed to Googlebot consequently supplies no supported crawl-rate setting for Google's crawler.

How does Bing describe the timing?

Bing's published explanation describes a value of 5 as dividing the day into five-second windows. Bingbot crawls at most one page in each window and may crawl none. The documented window model matters when interpreting a value as a limit on activity.

The same explanation says Bingbot manages crawl delays separately for subdomains with their own robots.txt files. Reviewing the file for each relevant hostname therefore follows the scope described by Bing.

FAQ

Can I use Crawl-delay to block ClaudeBot?

For a complete block, Anthropic's instructions give Disallow: / under User-agent: ClaudeBot. Its Crawl-delay example limits crawling activity.

Will adding Crawl-delay slow Googlebot?

Google's supported-field list explicitly excludes crawl-delay. Use that list when assessing which robots.txt instructions Google processes.

Sources

Check how a page reads to answer engines. Open the GEO score analyzer