Reference guide · technical-seo · Published 2026-08-15 · 4 min read
How crawling and indexing work
How Google crawling and indexing works: crawl, render, index, serve, discovery, exclusions, and what technical SEO controls.
- ·The four stages
- ·How discovery works
- ·What forces exclusion
The four stages
Searching engines move a page through stages. Knowing where your problem lives is most of technical SEO.
- Crawl: the engine finds and fetches a URL, reading the HTML response.
- Render: for modern engines the fetched page is placed in a queue to run JavaScript; the renderer executes scripts and builds the final DOM so the engine can read what a browser would show.
- Index: the engine analyses the rendered page, extracts content, and stores it with relevance and quality signals in the index.
- Serve: on a query, the engine ranks and returns the matching indexed pages.
Each stage is a checkpoint. A page can be crawled but never indexed, or indexed but never served if it sits too low in relevance.
How discovery happens
An engine finds pages through three routes:
- Links from pages already crawling, which is why internal and external links matter.
- The sitemap, which is a directed hint that a URL exists. See XML sitemaps.
- Discovery from existing URL patterns, the site root, or manual submission in search console.
There is no point at which a site "submits" pages to be ranked; the engine finds and evaluates them over time. A page with no links and no sitemap entry may never be discovered at all.
Resource limits
The engine has finite resource for any site, expressed as crawl budget. It will not crawl every URL of a huge site, and it renders a limited queue of pages that need JavaScript. Crawl frequency depends on site health and content value, not on a fixed schedule. For the practical levers, see crawl budget and site architecture.
What forces exclusion
Several mechanisms keep a page out of the index. They differ in how and where they act.
| Mechanism | Where it acts | Effect |
|---|---|---|
| robots.txt Disallow | Crawl stage | Stops the fetch, does not reliably deindex |
| meta robots noindex | Index stage, page must load | Requests exclusion from the index |
| X-Robots-Tag header | Index stage | Same control as meta, delivered in headers |
| Duplicate / canonical conflict | Index stage | Page consolidated or dropped against a canonical |
| 404 or 410 | Crawl stage | Page is gone and not indexed |
| Poor quality or thin content | Index stage | Treated as not index-worthy |
The key split: a robots.txt block stops the bot fetching a page, but if the page is indexed it stays indexed, because the bot can never read a noindex inside the page it was not allowed to fetch. To force exclusion, use noindex and keep the page fetchable. Compare robots.txt: what it can and cannot do with noindex and meta robots explained.
What technical SEO controls
Technical SEO is the discipline of keeping a site in the crawl-render-index-serve loop cleanly: fast responses for the crawl, renderable pages for the render stage, readable and unique content for the index stage, and correct canonical signals so the engine does not drop the wrong URL. This family of articles, the technical-seo hub, is where that stage-by-stage work lives, covering canonicals, robots, sitemaps, pagination, host choice, protocol, and architecture, all of which sit at one of these four gates.
Reference summary
- Crawl finds a URL; render builds the DOM; index stores it; serve returns it.
- Discovery runs on links, sitemaps, and pattern recognition.
- Exclusion runs at crawl or index stage, and the two mechanisms do not substitute.
- Health, value, and speed decide how often a site is re-crawled.