In short. X-Robots-Tag carries crawler instructions in HTTP response headers and supports Google indexing and preview rules for HTML and non-HTML resources.

Definition: X-Robots-Tag is an HTTP response header that carries crawler instructions for a URL, including instructions about indexing and search previews. In Google Search, it accepts the same rules as a robots meta tag, including for non-HTML resources such as PDFs.

Key facts

As of October 2026, Google's robots specification supports multiple X-Robots-Tag headers or a comma-separated list of rules in one header.
Google's robots.txt introduction explains that a web page blocked from crawling can still have its URL appear in search results.
Google's AI features documentation requires a page to be indexed and eligible for a search snippet to qualify as a supporting link in AI Overviews or AI Mode.

How is the header written?

An HTTP response header is information a server sends alongside the requested content. Google's header specification gives this instruction to exclude a resource from search results:

X-Robots-Tag: noindex

The noindex rule instructs Google to exclude the page, media or resource from search results. A robots meta tag places the equivalent instruction inside HTML; the header places it in the HTTP response for that URL.

Rules without a user agent apply to all crawlers that support them. A user agent identifies the crawler addressed by a rule. Google's example X-Robots-Tag: googlebot: nofollow addresses Googlebot and instructs it not to follow links on the page. The header name, user agent name and values are case-insensitive.

How does it apply to PDFs and images?

Google's implementation examples use server configuration to attach the header to non-HTML files. Its Apache example applies noindex, nofollow to PDF responses:

<Files ~ "\.pdf$">
Header set X-Robots-Tag "noindex, nofollow"
</Files>

Here, nofollow adds an instruction about following links to the indexing restriction. The example belongs in the site's root .htaccess or httpd.conf file. The same documentation provides an NGINX configuration and examples targeting image files or an individual static file. This makes the response header useful when a resource has no HTML document in which to place a robots meta tag.

What must crawlers be able to read?

Google's robots specification states that crawlers find these headers when they crawl the URL. The URL must therefore remain accessible to the crawler for its indexing and preview instructions to take effect. A robots.txt crawl block prevents Google from reading those instructions.

Google's AI features troubleshooting guidance recommends checking that preview controls are visible to Googlebot and allowing time for recrawling and processing. It points to the URL Inspection tool to examine the HTML Googlebot received. A change to the server configuration takes effect in Google's processing after Google retrieves and processes the changed response.

What changes for Google's AI search features?

A snippet is a preview of page content in search results. Google's rule definitions state that nosnippet prevents text snippets and video previews and prevents content from being used as direct input for AI Overviews and AI Mode. A static image thumbnail may remain visible.

The max-snippet rule sets a character limit for text previews and limits how much content may serve as direct input for those features. Google documents exceptions where publishers separately permit use, including through structured data or a licence agreement. These controls can be delivered through X-Robots-Tag.

FAQ

Can I combine several rules in one header?

Yes. Google's specification permits comma-separated rules, such as X-Robots-Tag: noindex, nofollow, and multiple header fields in the same response.

Which rule applies if preview instructions conflict?

Google applies the more restrictive rule. The same specification gives max-snippet:50 combined with nosnippet as an example: nosnippet applies.

Do other search engines interpret every rule identically?

Google's specification states that other search engines may treat these rules differently. The effects described here follow Google's documentation.

Sources

Check how a page reads to answer engines. Open the GEO score analyzer