In short. Cloudflare's behavior policies control recognized AI crawler categories, while AI Crawl Control provides individual crawler blocks and request metrics.
Definition: As of October 2026, Cloudflare documents Block AI bots as a control for verified AI training crawlers and some unverified bots with similar behavior. Its replacement policies separate Search, Agent, and Training activity.
Key facts
- Cloudflare's Block AI Bots documentation offers blocking across a domain or only on pages where Cloudflare detects ads.
- The AI Crawl Control setup guide requires a connected domain with traffic proxied through Cloudflare.
- The crawler management guide provides individual Allow and Block actions for crawlers.
- Cloudflare's robots.txt documentation describes compliance with that file as voluntary and directs access enforcement to AI Crawl Control.
Where is the setting, and what does it cover?
Cloudflare's Block AI Bots page gives this dashboard path for the behavior policies:
Security Settings > Configure AI bot policies
Search means collecting or indexing content for later answers. Agent means real-time activity on a person's behalf, including chat fetch bots. Training covers model training and fine-tuning. Each policy targets verified bots in its classification and additional unverified bots.
A training block across the zone, meaning the domain managed in Cloudflare, uses Training set to Block (on all pages). Block on pages with ads relies on automated ad detection. Allow (do not block) adds no blocking.
The same page gives Security Settings > Block AI bots as the legacy path and marks the older control for deprecation on 15 September 2026. Its dated transition notice includes mixed-purpose Search and Training crawlers in training blocks, while its legacy note still excludes them; the documentation leaves the legacy scope inconsistent.
How does individual crawler blocking work?
Cloudflare's setup guide places individual crawler controls in AI Crawl Control's Crawlers tab, where the Action column offers Block. On the free plan, detection recognizes well-known crawlers that identify themselves.
The crawler management documentation explains that blocking an individual crawler creates or updates a firewall rule called a WAF custom rule on the domain. That rule can be extended in WAF for path-based exceptions. These individual controls are separate from choosing a behavior policy in Security Settings.
How can the result be checked?
Cloudflare's setup guide describes a Metrics tab with breakdowns by crawler, response status code, hostname, and path. Free plans display the past 24 hours there.
The crawler management guide explains that unsuccessful requests can result from any rule or response error. An unsuccessful request count alone therefore cannot identify the blocking control responsible.
Related terms
- How do I enable Cloudflare AI Crawl Control?
- How do I allow one crawler in Cloudflare AI Crawl Control?
- Do Cloudflare's robots.txt controls also enforce request blocking?
- How do I check whether a CDN blocked an AI crawler?
FAQ
Can I block training while allowing search?
Cloudflare's behavior policies separate Search from Training. Mixed-purpose crawlers also used for training fall within its documented Training classification.
Is editing robots.txt enough to enforce a block?
Cloudflare's robots.txt documentation says the file expresses preferences and compliance is voluntary. It directs you to AI Crawl Control to enforce crawler-specific blocks.
Sources
- Cloudflare, "Block AI Bots", 1 July 2026
- Cloudflare, "Get started", 23 April 2026
- Cloudflare, "Manage AI crawlers", 28 July 2026
- Cloudflare, "robots.txt setting", 8 October 2026
Check how a page reads to answer engines. Open the GEO score analyzer