How do I set X-Robots-Tag on Cloudflare Pages?

Definition: Cloudflare Pages uses a plain text _headers file to attach HTTP response headers to matching static assets. An X-Robots-Tag: noindex header instructs supporting search engines to exclude the resource from search results.

The configuration and Google Search behaviour described here reflect documentation available in October 2026.

Key facts

• Cloudflare's custom headers documentation places _headers, without a file extension, in the project's static asset directory or build output directory.
• Google's robots specifications allow indexing directives in X-Robots-Tag, including for non-HTML resources such as PDFs.
• Google's noindex documentation requires the resource to remain accessible to the crawler, including through robots.txt, the file that controls crawler access.

Where does the file go, and what is the syntax?

Cloudflare's header rules consist of a URL or URL pattern followed by indented header lines. For a static deployment that should carry noindex across all paths, the file contains:

/*
  X-Robots-Tag: noindex

The asterisk matches all characters in the path. This broad rule also covers matching static responses on a custom domain, so a host-specific rule is appropriate when only the pages.dev address needs the directive.

For framework projects, the same documentation identifies public/ or static/ as common directories whose assets are copied into the final build output. _headers belongs in that copied output. Pages parses the file as configuration; the file itself is not served as a static asset.

How can rules target pages.dev URLs?

Cloudflare's documented indexing example supplies these host-specific patterns:

https://:project.pages.dev/*
  X-Robots-Tag: noindex

https://:version.:project.pages.dev/*
  X-Robots-Tag: noindex

:project and :version are named placeholders that match hostname components. The patterns cover the project hostname and hostnames with an additional component. Their hostname conditions leave custom-domain URLs outside these rules.

Matching rules accumulate. Cloudflare joins repeated header values with commas. A later rule should therefore not be treated as a replacement for every earlier matching rule.

How is the directive checked?

Google's noindex debugging guidance recommends URL Inspection to check the implementation and the Page Indexing report in Search Console to monitor pages where Googlebot, Google's crawler, extracts noindex. The intended response header is:

X-Robots-Tag: noindex

The same guidance explains that Google must crawl the page to read its headers. A robots.txt block prevents that access. Google also states that a page can remain in results until it is crawled again; URL Inspection provides a way to request a recrawl.

Related terms

• X-Robots-Tag
• noindex
• robots.txt
• How do I set different headers for Cloudflare Pages paths?
• Do Cloudflare Pages _headers rules apply to Functions responses?

FAQ

Does this work for Pages Functions?

Cloudflare's custom headers documentation excludes responses generated by Pages Functions, even when the URL matches. If you use Functions or server-side rendering, attach the header in the response-generating code.

Can I combine noindex with nofollow?

Yes. Google's robots specifications support comma-separated directives, such as X-Robots-Tag: noindex, nofollow. nofollow instructs Google not to follow links on the page; add it when that additional restriction is intended.

Sources

1. Cloudflare, "Headers", updated 25 August 2026
2. Google Search Central, "Robots meta tag, data-nosnippet, and X-Robots-Tag specifications", updated 24 March 2026
3. Google Search Central, "Block Search indexing with noindex", updated 10 December 2025

---

Check how a page reads to answer engines. Open the GEO score analyzer