In short. Claude-SearchBot supports Anthropic search indexing, with robots.txt controls and Crawl-delay support.

Definition: Claude-SearchBot is Anthropic's web crawler for improving the relevance and accuracy of search responses. It analyzes online content for search indexing.

Key facts

As of 17 October 2026, Anthropic's crawler privacy documentation states that blocking Claude-SearchBot prevents search indexing and may reduce a site's visibility and accuracy in search results.
The Robots Exclusion Protocol defines User-agent groups and path rules in a file named /robots.txt at the service's top level.
Anthropic publishes a crawler IP address feed containing creationTime, prefixes, and ipv4Prefix fields for checking crawler source addresses.

How does a website control Claude-SearchBot?

A robots.txt file supplies crawl instructions. A User-agent line identifies the crawler to which a group of rules applies; Disallow specifies paths excluded from crawling. Anthropic's privacy documentation instructs site owners to place blocking rules in the top-level robots.txt file and repeat the configuration for every subdomain to be excluded. Adapting its blocking example to the search crawler gives:

User-agent: Claude-SearchBot
Disallow: /

Under RFC 9309, crawler-name matching is case-insensitive. Multiple groups matching the same crawler are combined. The wildcard group, User-agent: *, applies when no crawler-specific group matches. Reviewing all matching groups therefore matters when editing an existing file.

For path rules, the RFC specifies the most specific matching path. Path matching should be case-sensitive, and an equivalent Allow rule should take precedence over Disallow. A rule covering the whole site should be reviewed alongside any more specific exceptions.

How can crawling be slowed or verified?

Anthropic's crawler guidance supports the non-standard Crawl-delay extension to limit crawling activity. Its example can be adapted to Claude-SearchBot:

User-agent: Claude-SearchBot
Crawl-delay: 1

This is an alternative configuration to the whole-site blocking example. Anthropic also states that a crawler with a source IP address on its published list comes from Anthropic.

The JSON feed groups network prefixes under prefixes; each listed range uses the ipv4Prefix field. A request's source address can be compared with those ranges. The feed identifies Anthropic's crawler infrastructure collectively, so the address check establishes the operator rather than a separate search-only address pool.

FAQ

Where should I put the rules?

RFC 9309 requires the lowercase filename /robots.txt at the service's top-level path. It specifies UTF-8 encoding and the text/plain media type.

Does robots.txt protect confidential pages?

The same RFC treats these rules as crawl instructions, separate from access authorization. For restricted pages, it recommends application-level security such as HTTP authentication; listing paths in robots.txt exposes them publicly.

What changes if I block Claude-SearchBot?

According to Anthropic, blocking it prevents search indexing of your content and may reduce your site's visibility and accuracy in user search results.

Sources

Check how a page reads to answer engines. Open the GEO score analyzer