In short. Check the matching security event and response to identify whether Cloudflare blocked or challenged an AI crawler request.
Definition: Checking a content delivery network (CDN) block means finding the crawler's request in security records and identifying the action applied to it. In Cloudflare, the useful evidence combines the request details, the responsible security rule and the response returned.
Key facts
- As of 17 October 2026, Cloudflare's AI Crawl Control documentation says unsuccessful requests can result from any rule or response error, including causes outside its crawler block setting.
- A web application firewall (WAF) filters incoming requests using rules. Cloudflare's integration documentation explains that a crawler set to Allow can still encounter blocking by earlier security rules.
- Cloudflare's Challenge Page documentation identifies challenge responses through the
cf-mitigatedheader with the valuechallenge.
Where does the investigation start?
Cloudflare's getting started guide requires the domain's traffic to pass through Cloudflare's proxy. On free plans, the Metrics tab shows the past 24 hours, so the investigation starts within that window.
In the dashboard, the account and domain are selected before opening AI Crawl Control. Cloudflare's traffic analysis guide describes filters for the date range, crawler, operator, hostname and path. Filtering to the affected page and crawler keeps the review tied to the reported access problem.
The Crawlers tab's Action column shows the current Allow or Block setting. In Metrics, All requests includes blocked requests and error responses; Allowed requests counts responses with status codes 200-299. These views establish the observed outcomes and current configuration before the individual event review.
Which security event explains the outcome?
Cloudflare's Security Events guide places the zone's events under Analytics, then the Events tab. Filters narrow the time period and request properties. Sampled logs lists the action taken and the security feature involved. Expanding the matching event reveals its details; Export event JSON saves the evidence for review.
A Ray ID is Cloudflare's request identifier. Its Ray ID documentation explains how to filter Security Events by this identifier and inspect associated request information. Matching the Ray ID alongside the time, path and client address helps connect a reported response to the event. Sampling can omit events, so a missing result calls for a narrower time range.
Cloudflare's WAF integration guide names the underlying rule AI Crawl Control and locates it under Security > Security rules, filtered by Custom rules. Comparing the event's rule with that rule distinguishes an AI Crawl Control block from another security rule's action.
How should the response be read?
Cloudflare's block response settings let paid plans configure a blocked crawler's response code and body. The documented code choices are 403 Forbidden and 402 Payment Required. The matched security event supplies the context for attributing that response to a CDN rule.
For a challenge, Cloudflare's response detection documentation specifies this response header:
cf-mitigated: challenge
The same documentation specifies text/html as the challenge's content type regardless of the requested resource. This marker identifies a Challenge Page returned in place of the anticipated response. An event with a Block action and an event with a Managed Challenge action therefore warrant different diagnoses.
Related terms
FAQ
Why does Allow still produce blocked requests?
Cloudflare's WAF integration guide says earlier custom rules can block an allowed crawler. You can inspect the matching event and review those rules before changing the crawler setting.
Does a crawler label verify its identity?
On free plans, Cloudflare's getting started guide says detection uses user-agent strings, the request headers identifying client software. If you use that plan, treat the label as identification based on the supplied header; Cloudflare documents more thorough detection through Bot Management detection IDs on upgraded plans.
Sources
- Cloudflare, "Manage AI crawlers", 28 July 2026
- Cloudflare, "AI Crawl Control with Cloudflare WAF", 3 August 2026
- Cloudflare, "Detect a Challenge Page response", 5 May 2026
- Cloudflare, "Get started", 23 April 2026
- Cloudflare, "Analyze AI traffic", 23 April 2026
- Cloudflare, "Security Events", 2 October 2026
- Cloudflare, "Cloudflare Ray ID", 15 October 2026
Check how a page reads to answer engines. Open the GEO score analyzer