In short. Enable Bot Preference Sync in Configure AI bot policies, set Training to Disallow, and save to add managed robots.txt directives.
Definition: As of 17 October 2026, Cloudflare's managed robots.txt uses Bot Preference Sync to generate crawler directives from AI bot policies. Cloudflare places managed content before an existing robots.txt file or serves a new file when none exists.
Key facts
- Cloudflare's robots.txt setting documentation lists managed robots.txt as available on all plans.
- Cloudflare's AI Crawl Control setup guide requires an account, a connected domain, and traffic proxied through Cloudflare for its crawler controls.
- The Directives documentation identifies a status card that shows whether Cloudflare manages the domain's robots.txt file.
Where is the setting enabled?
Cloudflare's implementation instructions start at Security Settings in the dashboard. The filter is Bot traffic, followed by Configure AI bot policies. For Training, the selected value is Disallow. Turning on Enable Bot Preference Sync and selecting Save applies the configuration.
Bot Preference Sync generates managed directives when a Search or Agent policy is Block or Block on pages with ads, or when Training is Disallow. With sync enabled, Training set to Disallow also enables blocking at Cloudflare's edge for AI training crawlers; mixed-use crawlers are excluded from that block.
The documentation's managed file includes this crawler-specific group:
User-agent: GPTBot
Disallow: /
The robots.txt file communicates access preferences, and compliance is voluntary. The blocking enabled by the Training policy operates through Cloudflare's request controls.
What happens to an existing robots.txt file?
Cloudflare's compatibility documentation uses an HTTP 200 response to identify an existing robots.txt file on the website's server. Cloudflare prepends its managed content and combines both parts into a single response.
The documentation illustrates this with crawlstop.com. Its original rules and sitemap remain after the managed section:
User-agent: *
Disallow: /lp
Disallow: /feedback
Disallow: /langtest
Sitemap: https://www.crawlstop.com/sitemap.xml
When no robots.txt file exists, Cloudflare creates and serves one containing managed Disallow rules for known AI crawlers. The managed response also includes Cloudflare's Content Signals Policy, which describes categories of permitted content use.
How is the configuration checked?
Cloudflare's Directives guide places verification under AI Crawl Control > Directives, after account and domain selection. The status card reports whether managed robots.txt is enabled.
The Robots.txt availability table provides a view across hostnames. Its Path column identifies each hostname's file, Status reports the HTTP result from pinging it, and Content Signals indicates whether usage directives are present. These fields support checking both management status and file availability.
If a file exists but requests fail, the guide recommends checking upstream firewall rules or other security settings that may block access. A high number of unsuccessful requests is a reason to investigate access to robots.txt.
Related terms
- robots.txt
- How do I enable Cloudflare AI Crawl Control?
- What Content Signals does Cloudflare add to managed robots.txt?
- Do Cloudflare's robots.txt controls also enforce request blocking?
FAQ
Do you need to delete your existing file?
Cloudflare's managed robots.txt documentation describes prepending managed content to your existing file. Its example retains the original path restrictions and sitemap after the managed section.
How can you block an individual crawler?
Cloudflare's setup guide directs you to AI Crawl Control > Crawlers. Select Block in the crawler's Action column.
Sources
- Cloudflare, "robots.txt setting", 8 October 2026
- Cloudflare, "Get started", 23 April 2026
- Cloudflare, "Directives", 23 April 2026
Check how a page reads to answer engines. Open the GEO score analyzer