What Content Signals does Cloudflare add to managed robots.txt?

Definition: Content Signals declare permitted uses of website content in robots.txt, the file that communicates crawler access preferences. As of 17 October 2026, Cloudflare documents a managed Content-signal line containing search=yes, ai-train=no, and the testing extension use=reference.

Key facts

• Cloudflare's managed robots.txt documentation shows search=yes, ai-train=no, use=reference in its managed example.
• Cloudflare's Content Signals announcement explains that an omitted signal expresses no permission or restriction through that signal.
• Cloudflare's content-use announcement describes use as an optional extension expressing what crawlers may retain and reuse.

What do the original fields mean?

Cloudflare's policy announcement defines the original vocabulary. search covers building an index and returning search results containing links and short excerpts. Its definition excludes AI-generated search summaries. ai-input covers feeding website content into models for real-time answers. ai-train covers model training or fine-tuning.

For these fields, yes permits collection for the corresponding purpose and no refuses it under the policy. The managed values therefore allow search use and refuse training use. The announcement explains why Cloudflare leaves ai-input unset: it avoids guessing the customer's preference. An omitted ai-input field leaves permission for that purpose unexpressed through Content Signals.

What does the additional use field mean?

Cloudflare's content-use announcement adds a separate dimension: what a crawler keeps and shares after access. It presents these levels in increasing order of permission:

• use=immediate permits interaction without storing or reusing content.
• use=reference permits indexing, excerpts, and links back.
• use=full permits summaries and reproduction.

The announcement adds use=reference to managed files. The managed example is:

User-Agent: *
Content-signal: search=yes, ai-train=no, use=reference
Allow: /

The use value accompanies the purpose fields on the same comma-separated line.

How can the served policy be checked?

The site's /robots.txt response is the place to inspect. Cloudflare's managed-file documentation says it places managed content before an existing file. Check the Content-signal: line as well as the policy comments beginning with #.

The same documentation distinguishes policy text from chosen values: qualifying Free-plan domains can receive the explanatory policy without expressing specific preferences. It also states that robots.txt compliance is voluntary. Cloudflare directs operators to AI Crawl Control for enforced crawler restrictions.

Related terms

• Content Signals
• robots.txt
• How do I enable Cloudflare's managed robots.txt?
• Do Cloudflare's robots.txt controls also enforce request blocking?

FAQ

Does search=yes include AI search summaries?

Cloudflare's policy definition excludes AI-generated search summaries from search. It defines model input for answers separately under ai-input.

Does seeing the policy mean I have selected ai-train=no?

The explanatory comments alone express no specific preference. Check your served file for a Content-signal: line containing ai-train=no.

Sources

1. Cloudflare, "robots.txt setting", updated 8 October 2026
2. Cloudflare, "Giving users choice with Cloudflare’s new Content Signals Policy", 24 September 2025
3. Cloudflare, "Your site, your rules: new AI traffic options for all customers", 1 July 2026

---

Check how a page reads to answer engines. Open the GEO score analyzer