noindex versus robots.txt

Crawling versus indexing

The single most important distinction in page removal is between crawling and indexing, because robots.txt and noindex operate on different stages of the pipeline. robots.txt is a crawl-level control: it tells a crawler *whether it may fetch a URL at all. The noindex directive, whether delivered via a robots meta tag or an X-Robots-Tag HTTP header, is an indexing-level (serving) control: it tells a crawler whether it may show the page in results*. Google's documentation is explicit that robots.txt permits and disallows crawling, and that recalling a URL via the Search Console Removals tool, noindex, or a 404 work differently at the indexing stage.

Because the two operate on different stages, their presence does not conflict only when you understand what each is for. robots.txt saves crawl budget by preventing requests. noindex spends a crawl (the crawler must fetch the page to read the directive) and then drops the page from the index. Neither is a synonym for the other.

The blocking trap

The failure mode that catches most sites is combining the two on the same URL. For a crawler to obey a noindex instruction, it must be able to fetch the page and read the tag inside it. If the same URL is disallowed in robots.txt, the crawler never fetches it, so it never sees the noindex; both directives are effectively dead for that URL. Google's guidance states plainly that if a page is disallowed from crawling, any indexing or serving rules on it "will not be found and will therefore be ignored."

The practical consequence is that a URL blocked in robots.txt can linger in the index as a bare link with no snippet, discovered from external links, even though you "blocked" it. Google also does not support noindex as a directive value *inside* robots.txt: it simply ignores it. Removing a page from search reliably means keeping it crawlable and serving the noindex on the page, not blocking it.

Which tool to use when

Choose the tool by the goal, not by habit:

• Reduce crawl load on a large class of valueless URLs (sort parameters, faceted filters, admin paths): robots.txt Disallow. It prevents the request entirely and is the strongest crawl-control mechanism.
• Definitively remove a specific known URL from the index: noindex, served on a crawlable page, or via X-Robots-Tag for non-HTML responses such as PDFs. Never also block it in robots.txt.
• Consolidate near-duplicate pages you must keep live: rel="canonical", where the canonical vs redirect guide or the canonicalisation edge cases apply.

A useful mental model is that robots.txt is about *budget and noindex is about result*. Use robots.txt when you want to stop the crawl, use noindex on a crawlable page when you want it out of results, and never stack the two on one URL. For HTML pages the directive lives in a robots meta tag (see noindex meta robots); for the header form see X-Robots-Tag.