In short. Cloudflare robots.txt directives are voluntary, while crawler Block actions enforce access restrictions and Training Disallow with Bot Preference Sync also enables training crawler blocking.
Definition: As of 17 October 2026, Cloudflare treats robots.txt directives as voluntary crawler instructions and enforces crawler-specific blocks through AI Crawl Control. Its Training Disallow policy with Bot Preference Sync enabled also activates blocking for AI training crawlers, excluding mixed-use crawlers.
Key facts
- Cloudflare's robots.txt setting documentation distinguishes voluntary directives from technical access restrictions and documents the combined Training Disallow and Bot Preference Sync behavior.
- The crawler management guide says selecting Block creates or updates a Web Application Firewall (WAF) custom rule to enforce that crawler's block.
- The Directives documentation compares current robots.txt directives with past requests when reporting violations.
- Cloudflare's WAF integration guide says crawlers set to Allow can still encounter blocks from other security rules.
What does the managed robots.txt file change?
A robots.txt file sits at a domain's root and tells crawlers which content they should access. Cloudflare's robots.txt setting documentation says managed content is prepended to an existing file that returns HTTP 200. If the site lacks a file, Cloudflare serves one containing managed Disallow rules for known AI crawlers.
The documentation's managed-file example includes this crawler-specific group:
User-agent: GPTBot
Disallow: /
These lines communicate a crawling restriction. A crawler's compliance with that file is voluntary.
The same documentation describes an associated enforcement setting: in Security Settings, under Bot traffic, Configure AI bot policies provides a Training policy. Selecting Disallow, enabling Bot Preference Sync, and saving adds managed directives and enables blocking at Cloudflare's network edge for AI training crawlers. Mixed-use crawlers are excluded from that block. The active policy therefore matters alongside the file's contents.
How does a crawler action enforce blocking?
AI Crawl Control provides tools for individual crawler access policies, as Cloudflare's product overview explains. The crawler management guide documents Allow and Block in the Actions column for each crawler.
Selecting Block creates or updates a WAF custom rule on the site's Cloudflare zone. A WAF filters incoming requests according to rules. Cloudflare's WAF integration documentation identifies the underlying rule as AI Crawl Control and locates it under Security > Security rules, filtered by Custom rules.
Allow also requires attention to the surrounding rules. The same integration guide explains that other WAF custom rules can block an allowed crawler. Conversely, earlier rules that skip WAF custom rules can bypass the AI Crawl Control block. Rule configuration and order affect the request's outcome.
How can the file and enforcement be checked separately?
Cloudflare's Directives guide describes a status card showing whether Cloudflare manages robots.txt. Its Robots.txt availability table reports file health across hostnames, including the HTTP response status and the presence of Content Signals, which describe content usage preferences.
The Robots.txt violations table identifies requested paths disallowed by the current file. Because it compares current directives with historical requests, adding a Disallow rule can cause earlier requests to appear as violations. The report therefore requires interpretation against the timing of changes.
The same guide directs site owners to the Crawlers tab to block a crawler, or to WAF for a path-specific security rule. A violation report supplies evidence for an access decision; the chosen policy or security rule performs enforcement. The crawler management guide also warns that unsuccessful request counts include errors and other rules, so that count alone cannot identify the AI Crawl Control block action.
Related terms
- robots.txt
- How do I enable Cloudflare's managed robots.txt?
- How do I enforce robots.txt directives with Cloudflare AI Crawl Control?
- How do I allow one crawler in Cloudflare AI Crawl Control?
FAQ
Does adding Disallow automatically block requests?
The directive itself relies on crawler compliance. Cloudflare's robots.txt setting documentation separately states that Training set to Disallow with Bot Preference Sync enabled also activates blocking for AI training crawlers, excluding mixed-use crawlers.
Can an allowed crawler still be blocked?
Yes. Cloudflare's WAF integration guide advises checking other WAF custom rules when a crawler configured as Allow still encounters blocking.
Does the violations table automatically enforce the file?
Cloudflare's Directives guide describes a report and separate actions: blocking the crawler in the Crawlers tab or creating a path-specific WAF rule.
Sources
- Cloudflare, "AI Crawl Control", 14 August 2026
- Cloudflare, "robots.txt setting", 8 October 2026
- Cloudflare, "Manage AI crawlers", 28 July 2026
- Cloudflare, "Directives", 23 April 2026
- Cloudflare, "AI Crawl Control with Cloudflare WAF", 3 August 2026
Check how a page reads to answer engines. Open the GEO score analyzer