In short. Review the Directives violations table, then apply a crawler block or a path-specific WAF custom rule and check rule precedence.
Definition: As of 17 October 2026, enforcement through Cloudflare AI Crawl Control means applying request-blocking rules to crawlers identified in its robots.txt violations report. Cloudflare's Web Application Firewall (WAF), which filters incoming requests using rules, applies the blocking.
Key facts
- Cloudflare's Directives documentation describes a violations table comparing current robots.txt directives with past crawler requests.
- Cloudflare's crawler management guide explains that selecting Block creates or updates a WAF custom rule for the domain.
- Cloudflare's WAF integration guide states that blocking several crawlers through AI Crawl Control uses one WAF custom rule.
Where is the compliance report?
After account and domain selection, the report path in Cloudflare's Directives documentation is:
AI Crawl Control > Directives > Robots.txt violations
Each row identifies the crawler and operator, the requested path, the exact directive restricting that path, and the request count for the selected period. Filters narrow the report by crawler, operator, use case or time frame.
The comparison uses the current file. Adding a new Disallow directive can therefore flag earlier requests that were permitted when they occurred. Reviewing the directive's change history alongside the selected period helps interpret those rows.
How is a blocking rule created?
For a crawler-wide block, Cloudflare's crawler management guide specifies Block in the crawler's Actions column. This creates or updates the domain's WAF custom rule. On the Free plan, detection relies on user-agent strings, the identifiers crawlers send with requests.
For a restriction limited to the reported path, the Directives documentation recommends a path-specific WAF rule. Cloudflare's custom-rule creation guide gives the dashboard procedure:
- Open Security rules, then Create rule > Custom rules, and enter a descriptive rule name.
- Under When incoming requests match, configure the crawler and path conditions using Field, Operator and Value.
- Under Then take action, choose Block to refuse matching requests.
- Select Deploy to activate the rule, or Save as Draft to retain it for review.
The same guide supports validation with Log on Enterprise plans. For other plans, it recommends starting with Managed Challenge, reviewing matches in Security Events, and then switching to Block.
What can interfere with enforcement?
Cloudflare's WAF integration guide locates the generated rule under Security > Security rules, filtered by Custom rules. Its name is AI Crawl Control.
The rule is appended after existing custom rules. Earlier Skip rules that bypass custom rules can interfere with blocking. Cloudflare recommends moving the AI Crawl Control rule to the top when troubleshooting blocked crawlers that still access content.
Direct edits can add path exceptions or hostname restrictions. These edits are preserved when crawler actions change, but the dashboard does not reflect direct WAF changes. An expression that AI Crawl Control cannot parse produces a warning with View rule in WAF.
Related terms
- robots.txt
- Do Cloudflare's robots.txt controls also enforce request blocking?
- How do I apply a Cloudflare WAF rule to one URL path?
- How does Cloudflare evaluate the order of custom WAF rules?
FAQ
Why does my new directive flag old requests?
Cloudflare's Directives documentation compares past requests with your current robots.txt file. A new restriction can change which historical requests appear in the report.
Does an unsuccessful request prove that my block worked?
Cloudflare's crawler management guide says unsuccessful requests can result from any rule or response error. Review the matching security rule when checking your enforcement.
Sources
- Cloudflare, "Directives", 23 April 2026
- Cloudflare, "Manage AI crawlers", 28 July 2026
- Cloudflare, "AI Crawl Control with Cloudflare WAF", 3 August 2026
- Cloudflare, "Create a custom rule in the dashboard", 8 October 2026
Check how a page reads to answer engines. Open the GEO score analyzer