In short. Claude-User retrieves website content for Claude user queries, and Anthropic documents robots.txt rules for blocking that access.

Definition: As of 17 October 2026, Claude-User is Anthropic's agent for retrieving website content in response to questions from Claude users. Anthropic documents robots.txt controls that let website owners prevent this user-directed retrieval.

Key facts

Anthropic's crawler privacy documentation says disabling Claude-User prevents content retrieval in response to a user query and may reduce visibility in user-directed web search.
The Robots Exclusion Protocol specification defines robots.txt as a file of crawler access rules, located at the service's top-level path.
Anthropic's published crawler address list contains network ranges under the field ipv4Prefix.

How does Claude-User differ from Anthropic's other bots?

Anthropic's privacy documentation assigns Claude-User to requests initiated by users asking Claude questions. Claude may access a website through this agent when answering a question.

The same document assigns model-training collection to ClaudeBot and search-result improvement to Claude-SearchBot. It describes restricting ClaudeBot as a signal to exclude future website materials from training datasets. Disabling Claude-SearchBot prevents indexing for search optimization. These separate names allow website owners to express access preferences for each documented purpose.

How does a website opt out of Claude-User retrieval?

A robots.txt file communicates crawler access preferences. Anthropic's blocking instructions say its bots honor these directives and require the file to be updated to opt out. Adapting its whole-site blocking example to Claude-User gives:

User-agent: Claude-User
Disallow: /

The User-agent line selects the agent, and Disallow: / restricts access across the site's paths. The instructions require a robots.txt file for every subdomain whose access should be restricted.

RFC 9309, the Robots Exclusion Protocol specification, requires the lowercase path /robots.txt, UTF-8 encoding and the text/plain media type. It also requires matching groups for the same agent to be combined, with the most specific matching path rule taking precedence. Reviewing existing groups and exceptions is therefore part of checking the effective rule.

How can a website check the request's origin?

Anthropic's privacy documentation says a crawler source address appearing in its linked list indicates that the crawler comes from Anthropic. Comparing a request's source address with the ranges in that JSON list provides the documented origin check.

The same privacy documentation warns that blocking source addresses can interfere with reading robots.txt and make opt-out handling unreliable. Its prescribed opt-out method remains the robots.txt change.

FAQ

Can I block Claude-User even though a person initiates the request?

Yes. Anthropic's crawler documentation explicitly lets you disable Claude-User retrieval in response to user queries through robots.txt.

Which agent controls collection for training?

The same documentation assigns that role to ClaudeBot. Use the agent name corresponding to the activity you want to restrict.

Does robots.txt protect confidential content?

RFC 9309's security guidance calls for application-level security measures, such as HTTP authentication, to control access. Paths listed in robots.txt are public.

Sources

Check how a page reads to answer engines. Open the GEO score analyzer