In short. ClaudeBot collects potential Anthropic training content under robots.txt access controls.

Definition: As of 17 October 2026, ClaudeBot is Anthropic’s crawler for collecting web content that may contribute to model training, according to its crawler guidance.

Key facts

Restricting ClaudeBot signals exclusion of future materials from Anthropic’s training datasets, according to its crawler guidance.
Anthropic’s published address list supplies IPv4 network prefixes for its crawlers.
The IETF’s Robots Exclusion Protocol defines robots.txt as crawl instructions and states that these rules are separate from access authorization.

What does ClaudeBot access control cover?

Anthropic’s crawler guidance assigns training collection to ClaudeBot, search indexing to Claude-SearchBot, and user-requested retrieval to Claude-User. Each has its own access control.

How is ClaudeBot blocked?

Anthropic provides this whole-site blocking example, to apply separately on each relevant subdomain:

User-agent: ClaudeBot
Disallow: /

The protocol’s access method places the file at the service’s top-level /robots.txt path, in lowercase. It requires UTF-8 encoding and the text/plain media type. A robots.txt file located in a content subdirectory does not occupy that required location.

The user-agent rule selects a crawler by its product token, the name used to find applicable instruction groups. Here that token is ClaudeBot.

How can a website check its configuration?

A review starts with the existing robots.txt groups. Under the protocol’s matching rules, matching crawler names is case-insensitive, and groups matching the same name are combined. The User-agent: * group applies when no specific group matches. Consequently, reviewing only the wildcard group can miss the rules that apply to ClaudeBot.

The path-matching rules use the most specific matching path. Where equivalent Allow and Disallow rules conflict, the protocol recommends using Allow. These details matter when a blocking group contains exceptions. The protocol also recommends case-sensitive matching for paths, so the spelling of a directory matters when reviewing an exception.

Anthropic’s crawler address feed is a JSON object with creationTime and prefixes fields. Its network entries use the key ipv4Prefix. These are network ranges: a request’s source address can be checked for membership in a listed range. Anthropic’s guidance identifies this check as evidence that a crawler comes from Anthropic.

FAQ

Can you slow ClaudeBot instead of blocking it?

Anthropic’s guidance supports the non-standard Crawl-delay robots.txt extension to limit crawling activity.

Does robots.txt protect confidential content?

The protocol’s security considerations explain that robots.txt rules are publicly accessible. For content requiring protection, the document points to application-level security, such as HTTP authentication.

Sources

Check how a page reads to answer engines. Open the GEO score analyzer