Reference guide · technical-seo · Published 2026-08-15 · 4 min read
noindex and meta robots explained
noindex and meta robots guide: the robots directive table, X-Robots-Tag versus meta, and noindex versus robots.txt Disallow.
- ·The directives table
- ·Meta versus header
- ·noindex versus robots.txt
What meta robots is
Meta robots is a per-page instruction read by search engines and listed in the <head>. The two directives that matter most are noindex, which asks engines not to include the page in their index, and nofollow, which asks engines not to follow links on the page. Both can appear together in one value.
<meta name="robots" content="noindex,follow" />The lowercase directives are separated by commas and there is no space after the comma. A page can carry several directives at once, for example noindex,nofollow,noarchive.
The directives table
| Directive | Effect |
|---|---|
noindex | Do not show this page in search results |
nofollow | Do not follow links on this page |
noarchive | Do not store a cached copy |
nosnippet | Do not show a text snippet |
notranslate | Do not offer a translation of the page |
index / follow | Allow indexing / link following (the default) |
Use index,follow rarely; it is the default behavior and mostly appears when you want to be explicit or when other directives might inherit.
Meta tag versus HTTP header
The same directives can be sent as the X-Robots-Tag HTTP header instead of an in-page meta tag. The header is useful when you cannot edit a cached or generated page body, when you manage directives at the server or CDN layer, or when you want to apply noindex to non-HTML files such as PDFs where no <head> exists.
| Delivery point | Where it lives | Typical use |
|---|---|---|
<meta name="robots"> | Inside the HTML <head> | Per-page control on editable pages |
X-Robots-Tag header | HTTP response header | PDFs, assets, server-wide rules |
HTTP/1.1 200 OK
X-Robots-Tag: noindex, nofollowBoth routes require the page to be fetchable. A bot has to download the response to read either instruction.
noindex versus robots.txt Disallow
This distinction drives a lot of indexing confusion.
- A robots.txt
Disallowstops the bot from fetching the URL. It does not tell the engine the page should be excluded from the index, and a page can still be indexed if it is discovered by another route, such as links or a sitemap. Because the bot cannot fetch the page, it can never read a meta noindex inside it, so the indexing state is unresolved. - A
noindexdirective actively requests removal from the index. It requires the page to be fetchable, then tells the engine exactly what to do.
In practical terms:
# robots.txt - blocks fetch, not indexing
User-agent: *
Disallow: /old-promo/<!-- meta robots - requests no index, page must be fetchable -->
<meta name="robots" content="noindex" />If you want a page out of the results, use noindex and keep it crawlable, rather than relying on Disallow. If you want to stop wasting crawl budget on an area, Disallow is your tool. The two have different jobs; choose the one that matches your goal. See robots.txt: what it can and cannot do.
How to apply noindex (ordered)
- Confirm the page returns a fetchable status, typically 200, not a robots block.
- Add
<meta name="robots" content="noindex,follow" />to the<head>, or set theX-Robots-Tag: noindexheader. - Let the URL be re-crawled; in Search Console, request indexing on the page.
- Verify the header or tag in a fetch that shows raw HTML or response headers.
- Leave the URL reachable so the engine can see the instruction and honour it.
A canonical tag pairs naturally with indexing control; combine them only when the decision is about consolidation rather than hiding. See canonical tags explained and how exclusion fits the wider pipeline in how crawling and indexing work.