Using the x-robots-tag HTTP header

What the x-robots-tag header is

X-Robots-Tag is an HTTP response header that carries the same indexing directives as the robots meta element, but set on the server side. Instead of living inside a page's HTML, it arrives with every response:

HTTP/1.1 200 OK
Content-Type: application/pdf
X-Robots-Tag: noindex, nofollow

That one line tells search engines not to index the returned resource and not to follow its links. Because it is an HTTP header, it can be applied to any resource, including files that have no HTML head to carry a meta tag: PDFs, images, videos, API responses and feeds.

Directives

X-Robots-Tag accepts the same values as the meta robots element, plus a few Google-specific ones:

You can combine values in one header: X-Robots-Tag: noindex, nofollow. More than one value with different scopes (like none and max-snippet) can also be declared by sending the header multiple times, one value each.

x-robots-tag vs meta robots vs robots.txt

The three indexing controls overlap and interact:

• robots.txt governs crawling: it tells the crawler which URLs it may request. It cannot force a noindex; a disallow keeps the URL unvisited, which leaves it out of the index, but a page linked elsewhere can still get indexed via other signals. It is a crawl directive, not an index directive.
• meta robots is in the page HTML: a per-page directive a crawler honours after fetching the page.
• x-robots-tag is in the HTTP response: a per-response directive, which is why it works for non-HTML resources, and can be set server-wide with an edge rule, a single directive applying to many URLs.

Google's rule when the header and the meta tag disagree: it acts on the most restrictive value. A page with meta robots: noindex in the body and a X-Robots-Tag: noindex header carries the restriction twice, which is harmless; it is the situations where one says can and the other says cannot that the restrictive one wins.

Why you would use the header

1. Non-HTML files: PDFs, DOCX, images, audio. A PDF has no head, so the header is the only per-resource directive.
2. Response-level logic: the web server or CDN can set the header for every file of a type in one rule, without touching HTML. For example, add X-Robots-Tag: noindex on every /downloads/*.pdf path at the edge.
3. Per-page control with payloads that do not care: CDNs, Workers, edge functions and .htaccess/nginx configs can all emit it. The header works even when the page itself is cached.

The shared pattern with WordPress or static hosting: set the noindex on content you want discoverable but not directly in the results (thank-you pages, filters, temporary paths) while the URL stays crawlable.

Setting it: three examples

Apache (.htaccess)

<Files "*.pdf">
  Header set X-Robots-Tag "noindex"
</Files>

nginx

location ~* \.pdf$ {
  add_header X-Robots-Tag "noindex";
}

Cloudflare / edge: a redirect or transform rule adds X-Robots-Tag: noindex on the passthrough.

Run them per environment. The rule does not send robots.txt content; it sends only the header, so the resource stays crawlable but deindexed.

Verify

Check the response directly:

• curl -I https://example.com/file.pdf shows the header.
• The Search Console URL inspection tool reports the robots state it saw for that URL, including header values.
• A stale edge cache can serve a header that no longer applies, so the check needs a cache-busted request after changes.

The URL inspection flow and the canonical tag guide complete the deindexing story, because the header alone does not fight canonical choices.