How can CloudFront apply a robots header only to a selected URL path?

Definition: Amazon CloudFront can apply an X-Robots-Tag response header to a selected URL path by placing the header in a response headers policy and attaching that policy to the cache behavior whose path pattern matches the request. CloudFront applies the policy to responses for requests that match that cache behavior.

Key facts

• Google documents X-Robots-Tag as an HTTP response header for a given URL, and says that rules available in a robots meta tag can also be specified in the header (Google Search Central's specification).
• AWS says that a response headers policy specifies headers that CloudFront adds or removes, and that a policy can be attached to one or more cache behaviors (Amazon CloudFront's response policy guide).
• A CloudFront path pattern selects a cache behavior because the requested path is compared with listed patterns and the first match determines the behavior (AWS cache behavior settings).
• AWS's console procedure places the Response headers policy selector inside the cache behavior settings, followed by Save changes (AWS policy creation guide).
• Headers added by a response headers policy are sent in responses to viewers. AWS says that headers such as Cache-Control added by the policy affect the viewer response and do not change how CloudFront caches the requested object.

How does CloudFront select the path?

A cache behavior is a set of CloudFront settings for a URL path pattern. AWS's example uses images/*.jpg. CloudFront compares an end user's requested path with the path patterns in the order in which the cache behaviors are listed. The first matching pattern determines which behavior is applied.

The default cache behavior has the path pattern *. It is processed last, and CloudFront uses it when a request does not match another cache behavior. Place the selected-path behavior before a broader behavior that could also match. AWS documents that a pattern can include a leading slash, that * matches zero or more characters, and that ? matches exactly one character.

CloudFront's path evaluation ignores query strings and cookies.

How do I create the response headers policy?

Create a custom response headers policy in the CloudFront console. In the policy form, AWS provides a Custom headers panel for adding headers to the policy. Add the header and value used for the crawler instruction:

X-Robots-Tag: noindex

Google's specification uses this header to instruct crawlers not to index a page. It also documents nofollow, nosnippet, and other robots rules for the same header. A rule without a user agent applies to all crawlers according to that specification.

How do I attach it only to the selected behavior?

Open the CloudFront Distributions page and choose the distribution. On Behaviors, select the cache behavior for the selected path and choose Edit. In Response headers policy, choose the policy, then choose Save changes. If the path does not have its own behavior, AWS's procedure allows you to choose Create behavior and configure the path pattern there.

The policy applies to requests that match that behavior. AWS says it modifies both responses served from the cache and responses forwarded from the origin.

For an API or an infrastructure configuration, AWS identifies the association with the ResponseHeadersPolicyId field inside a cache behavior. The response headers policy ID is created first, then supplied when the distribution is updated.

What should you check before relying on the header?

Google says that X-Robots-Tag instructions are found when a URL is crawled. If robots.txt disallows crawling of the URL, the indexing or serving rule in the response is unavailable to the crawler. Check the path pattern order and request the selected URL through CloudFront, then inspect the HTTP response for the header.

A browser caching header has a separate effect: AWS limits a Cache-Control header added by a response headers policy to the viewer response, so that header does not control CloudFront's cache duration.

Related terms

• X-Robots-Tag
• Which value wins when CloudFront and the origin set the same custom response header?
• What happens when a CloudFront policy removes and adds the same response header?
• Does adding Cache-Control through a CloudFront response policy change edge caching?

FAQ

Can a broad default behavior override the selected path behavior?

No. CloudFront processes the listed cache behaviors in order and uses the first matching path pattern. The default behavior is processed last, so a selected-path behavior must be placed before a broader matching behavior.

Can the policy add X-Robots-Tag to cached responses?

Yes. AWS says that CloudFront modifies the headers in responses served from the cache as well as responses forwarded from the origin when the request matches the behavior.

Does X-Robots-Tag replace robots.txt?

They operate at different points. Google says the header is found when the URL is crawled, while a robots.txt disallow prevents the crawler from finding the header's indexing or serving rule.

Does the policy change CloudFront's edge cache duration?

The AWS note specifically says that a Cache-Control header added by a response headers policy affects the viewer response and does not affect how CloudFront caches the requested object.

Sources

1. Amazon Web Services, "Add or remove HTTP headers in CloudFront responses with a policy", accessed 17 October 2026
2. Amazon Web Services, "Cache behavior settings", accessed 17 October 2026
3. Amazon Web Services, "Create response headers policies", accessed 17 October 2026
4. Google Search Central, "Robots meta tag, data-nosnippet, and X-Robots-Tag specifications", 24 March 2026

---

Check how a page reads to answer engines. Open the GEO score analyzer