In short. Select Allow for the intended crawler in AI Crawl Control, then review conflicting security rules and the crawler’s response metrics.
Definition: As of October 2026, Cloudflare AI Crawl Control provides an Allow action for individual AI crawlers in its Crawlers table. Selecting Allow permits access through that control, while other security rules can still block the crawler's requests.
Key facts
- Cloudflare's setup guide requires a connected domain with traffic proxied through Cloudflare.
- The crawler management documentation instructs site owners to select Allow in the Actions column for the chosen crawler.
- Cloudflare's WAF integration documentation states that the AI Crawl Control rule includes blocked bots; an allowed crawler can still encounter another security rule.
- The traffic analytics documentation defines Allowed requests as requests receiving a successful response with status 200-299.
Where is the per-crawler Allow setting?
Cloudflare's Manage AI crawlers guide starts with selecting an account and domain in the dashboard, then opening AI Crawl Control. The Crawlers table identifies each crawler and its operator, the company responsible for it. Name and Operator filters help locate the intended crawler.
For the chosen crawler, selecting Allow in the Actions column sets its access policy. Each crawler has an individual action, so other crawler rows can retain Block. The crawler-directive file robots.txt can also be enforced alongside Allow, according to the same guide. Enforcement remains a separate choice.
Cloudflare's analytics guide identifies the Crawlers tab as the place for Allow and Block controls. Its Action column shows the current access control setting, which provides a way to review the selected row after changing it.
What does Allow change for incoming requests?
A Web Application Firewall (WAF) filters incoming requests using rules. Cloudflare's WAF integration guide explains that AI Crawl Control uses a WAF custom rule for the crawlers selected for blocking. Allow leaves a crawler outside that rule's blocked selection. Other security rules continue to apply.
If an allowed crawler remains blocked, the same guide directs site owners to inspect earlier WAF custom rules. The AI Crawl Control rule is named AI Crawl Control. Its documented dashboard location is:
Security > Security rules
Filtering by Custom rules locates it. A separate rule that blocks all bots can conflict with the intended crawler policy. Cloudflare advises identifying and modifying conflicting custom rules. Placing the AI Crawl Control rule earlier does not itself give an allowed crawler an exemption from another blocking rule.
How can the request outcome be checked?
Cloudflare's traffic analytics guide documents View Metrics in each crawler's actions menu. Selecting it filters the Metrics tab to that crawler. Date range, hostname and path filters narrow the requests under review.
The Status code distribution chart separates successful responses, redirects, client errors and server errors. The Allowed requests metric counts responses with status 200-299. This measures request outcomes alongside the access policy shown in the crawler row.
The crawler management guide states that unsuccessful requests can result from any rule or response error. An unsuccessful request count therefore requires inspection of the applicable rules and responses before attributing it to the crawler's Action setting.
Related terms
- How do I enable Cloudflare AI Crawl Control?
- How do I check whether a CDN blocked an AI crawler?
- Do Cloudflare's robots.txt controls also enforce request blocking?
- How do I apply a Cloudflare WAF rule to one URL path?
FAQ
Can I allow one crawler while blocking others?
Yes. Cloudflare's crawler management guide provides a separate Allow or Block choice for each crawler. You can select Allow for the intended crawler and leave other rows set to Block.
Why is my crawler still blocked after I select Allow?
Cloudflare's WAF troubleshooting guidance points to earlier custom rules that can block allowed crawlers. You can inspect those rules under Security > Security rules, filtered by Custom rules.
Where can I compare the setting with successful requests?
Cloudflare's analytics documentation lists the current setting in the Crawlers table's Action column. You can select View Metrics for that crawler to inspect Allowed requests and the response code distribution.
Sources
- Cloudflare, "Get started with Cloudflare AI Crawl Control", 23 April 2026
- Cloudflare, "Manage AI crawlers", 28 July 2026
- Cloudflare, "AI Crawl Control with Cloudflare WAF", 3 August 2026
- Cloudflare, "Analyze AI traffic", 23 April 2026
Check how a page reads to answer engines. Open the GEO score analyzer