How can an X-Robots-Tag header target a named crawler?

Definition: An X-Robots-Tag user-agent prefix is the crawler name placed before a colon in an HTTP response header, followed by one or more robots rules. Google's robots metadata reference shows googlebot: nofollow and otherbot: noindex, nofollow as conditional examples.

Key facts

• An X-Robots-Tag is an HTTP response header for a URL, and a rule available in a robots meta tag can also be specified in that header, according to Google's reference.
• The user-agent prefix is optional. A rule without one applies to all crawlers, and Google states that the header, user-agent name, and values are case-insensitive.
• Multiple X-Robots-Tag headers can be used, or rules can be written as a comma-separated list.
• HTTP field names are case-insensitive. When a field name is repeated, HTTP combines its values in order with commas, as described in RFC 9110.

What does the named-crawler syntax look like?

The prefix is the crawler name before the colon. Google documents this response pattern:

HTTP/1.1 200 OK
X-Robots-Tag: googlebot: nofollow
X-Robots-Tag: otherbot: noindex, nofollow

The first field line assigns nofollow to googlebot. The second assigns both noindex and nofollow to otherbot. The same documentation says that the user agent may appear before the rules, and that rules without a user agent are valid for all crawlers. The comparison is case-insensitive, so the documented token and rule names do not depend on letter case.

How do multiple headers and rules work?

Google states that multiple X-Robots-Tag headers can be combined in one response, and that a comma-separated list can express several rules. RFC 9110 describes a repeated HTTP field name as an ordered list whose field-line values are concatenated with a comma. It also says that a recipient may combine same-name field lines into one field line when the field definition permits that list form.

For a single crawler, the rule list can therefore be expressed on one field line:

X-Robots-Tag: googlebot: noindex, nofollow

Google also documents conflicting robots rules and states that the more restrictive rule applies. Its example compares max-snippet:50 with nosnippet, with nosnippet taking effect.

What can the header control on a website?

The X-Robots-Tag carries the same rule family that Google documents for robots meta tags. It is useful for responses that are not HTML, including PDF, video, and image files, where an HTML meta tag cannot be placed. Google says that the header can be added through a web server's configuration files, including Apache .htaccess and httpd.conf files and an NGINX .conf file. The page also documents regular-expression support in this configuration context.

The response header belongs to a particular URL. Google says that robots meta tags and X-Robots-Tag headers are discovered when a URL is crawled. If robots.txt disallows that URL, the indexing or serving rules in the response are not found and are ignored.

How should the response be checked?

The relevant artefact is the HTTP response for the URL. A check should record the field name, the optional crawler prefix before the colon, and the rules after it. RFC 9110 defines a field name as the label for its field value and specifies that field names are case-insensitive. Google’s documented examples provide the values to compare: googlebot, otherbot, nofollow, noindex, and comma-separated rule lists.

Related terms

• X-Robots-Tag
• Meta robots
• robots.txt User-agent directive
• Googlebot

FAQ

Can you apply one X-Robots-Tag rule to every crawler?

Yes. Omit the user-agent prefix. Google states that a rule specified without a user agent is valid for all crawlers.

Can you put several rules after one crawler name?

Yes. Google documents comma-separated rules, such as otherbot: noindex, nofollow, and also documents multiple X-Robots-Tag headers in one response.

Can you use the header for a PDF or image?

Yes. Google documents the X-Robots-Tag for non-HTML resources, including PDF, video, and image files.

Sources

1. Google Search Central, "Robots meta tag, data-nosnippet, and X-Robots-Tag specifications", 24 March 2026
2. IETF, "HTTP Semantics", June 2022

---

Check how a page reads to answer engines. Open the GEO score analyzer