How can images be excluded from Claude's search partners?

Definition: To exclude images or video from Claude web search, block their file URLs in robots.txt for the crawler that searches the site. Anthropic's guidance says this media-specific option applies because its search partners index only images and videos their bots are allowed to crawl.

Key facts

• Anthropic's content removal guidance marks robots.txt disallow rules as applicable to images and video and tells site owners to block the media files.
• RFC 9309 defines a robots.txt group as one or more User-agent lines followed by rules, and says a crawler with no matching specific group uses User-agent: * when that group exists.
• Anthropic's crawler documentation identifies Claude-SearchBot as the bot that crawls the web to improve search result quality and says disabling it prevents Anthropic from indexing content for search optimization.
• Anthropic lists removing content, password protection, noindex, robots.txt rules for media, and a separate Anthropic bot opt-out as different controls with different scopes.

Which robots.txt rule blocks an image path?

RFC 9309 calls the value on a User-agent line a product token. A path-specific group for Anthropic's documented search crawler can therefore be written as:

User-agent: Claude-SearchBot
Disallow: /media/

The rule asks that crawler to avoid URLs whose path starts with /media/. Replace /media/ with the directory or exact path that serves the files. RFC 9309 says matching starts at the first octet of the path and that the most specific matching rule is used.

The group applies to the host serving the media. RFC 9309 requires the rules to be available at a file named /robots.txt in the service's top-level path. If the images use a separate media host, that host has its own robots.txt location.

Which Claude crawler does the rule target?

Anthropic's crawler documentation lists three crawler names with separate purposes. ClaudeBot collects public web content for model development, Claude-User retrieves web content for user-directed requests, and Claude-SearchBot crawls the web to improve search result quality. A group naming Claude-SearchBot therefore targets the documented search crawler, while a group naming ClaudeBot targets a different crawler.

Anthropic says its bots honor industry-standard robots.txt directives. Its whole-site example for a bot uses this form:

User-agent: ClaudeBot
Disallow: /

That example covers the named bot across the site, with the file placed in the top-level directory. Anthropic also says to place the file on every subdomain for which an opt-out is wanted. A media-path rule keeps the scope at the selected path instead of using the root path.

How does this differ from noindex or removing the file?

Anthropic's content removal guidance says removing content from the site is the best way to ensure that it will not appear in Claude outputs when Claude searches the web. It lists noindex as applicable to all content types and says the tag tells its partners not to index the content, while the content can still be linked to and visited.

The same guidance assigns robots.txt disallow rules specifically to images and video. Use that path control when the file should remain available to visitors while the crawler is asked to avoid it. RFC 9309 describes robots rules as instructions for crawler access and states that they are not access authorization. Anthropic's public media guidance does not list separate user-agent tokens for each search partner, so the partner's own documented token is needed when a partner-specific group is available.

Related terms

• Claude-SearchBot
• robots.txt Disallow directive
• robots.txt User-agent directive
• robots.txt

FAQ

Should an image use noindex or robots.txt?

Anthropic assigns noindex to all content types and robots.txt disallow rules to images and video. For a media-specific crawler access restriction, use the media URL path in robots.txt.

Does Disallow delete the image?

No. The file stays on the site. RFC 9309 describes robots rules as crawler access instructions and says they are not access authorization.

Does a Claude-SearchBot group also control ClaudeBot?

No. The two names identify separate Anthropic crawlers. Give each crawler its own group when each needs a different path policy.

Sources

1. Anthropic, "Report, block, and remove content from Claude", 4 September 2026
2. Anthropic, "Does Anthropic crawl data from the web, and how can site owners block the crawler?", 7 April 2026
3. RFC Editor, "RFC 9309: Robots Exclusion Protocol", September 2022

---

Check how a page reads to answer engines. Open the GEO score analyzer