In short. Block ClaudeBot with Disallow: / and use separate Allow: / groups for Claude-SearchBot and Claude-User to permit their search and retrieval functions.
Definition: As of 17 October 2026, Anthropic documents separate robots.txt controls for its training crawler ClaudeBot, search crawler Claude-SearchBot and user-request agent Claude-User. A site can block ClaudeBot while permitting the other agents to access its content for their respective search and retrieval functions.
Key facts
- Anthropic's crawler privacy documentation assigns different functions to ClaudeBot, Claude-SearchBot and Claude-User and says its bots honor robots.txt directives.
- The IETF's Robots Exclusion Protocol specifies that a crawler follows its matching user-agent group; the wildcard group applies when there is no matching group.
- Anthropic's web search help describes web fetch: when web search is enabled, Claude can retrieve page content from a supplied URL.
What should the robots.txt file contain?
A robots.txt file expresses crawler access rules. A User-agent line identifies the crawler to which the following rules apply. Disallow: / blocks all paths, while Allow: / permits them.
Anthropic's crawler instructions give a ClaudeBot block example. Combining that example with explicit permissions for the other documented agents produces these separate groups:
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
These groups belong in the site's existing robots.txt file in its top-level directory. The same documentation requires the opt-out on every subdomain concerned. A rule on the main website therefore needs a corresponding configuration on each subdomain where the same policy is intended.
The example permits all paths for Claude-SearchBot and Claude-User. Any intended path restrictions for these agents belong in their respective groups.
Why allow Claude-User as well as Claude-SearchBot?
Anthropic's bot descriptions associate Claude-SearchBot with improving search responses and indexing content for search. Claude-User accesses websites in response to user requests. Disabling Claude-User prevents that user-requested retrieval; disabling Claude-SearchBot prevents the indexing described in the documentation.
Allowing both agents preserves permission for both documented functions. Restricting ClaudeBot signals that the site's future materials should be excluded from Anthropic's model-training datasets.
How do existing rules affect this configuration?
Under RFC 9309, the standard defining the Robots Exclusion Protocol, a matching named group takes precedence over the fallback User-agent: * group. A wildcard block therefore needs particular attention when the file lacks explicit groups for Claude-SearchBot or Claude-User.
The same standard requires rules from multiple groups matching the same agent to be combined. Adding a new group leaves existing matching rules in play. Within the combined rules, the most specific matching path determines access. A blanket allow at / can coexist with a more specific block on a directory.
The file also needs to be served at /robots.txt as UTF-8 text with the text/plain media type, as specified by the standard. Reviewing the complete file, including duplicate agent groups and path restrictions, establishes what the configuration permits.
Related terms
FAQ
Can I block training crawling while allowing search access?
Yes. Anthropic's crawler documentation documents separate agents for these functions. You can disallow ClaudeBot and allow Claude-SearchBot, with a separate choice for Claude-User retrieval.
Does a wildcard block also block the search agents?
Under the Robots Exclusion Protocol, it applies when there is no matching named group. You can add explicit groups for Claude-SearchBot and Claude-User, then review any other groups matching those agents.
Sources
- Anthropic, "Does Anthropic crawl data from the web, and how can site owners block the crawler?", 7 April 2026
- IETF, "RFC 9309: Robots Exclusion Protocol", September 2022
- Anthropic, "Enable and use web search", accessed 17 October 2026
Check how a page reads to answer engines. Open the GEO score analyzer