Crawl budget and site architecture

What crawl budget is

Crawl budget is the allocation used by a search engine when it decides how often and how many URLs from your site to fetch within a time window. It is not a fixed number you can read off a control panel. It is a working equilibrium influenced by server response, site health, content value, and how quickly the engine can chew through pages.

Two things feed the picture. Crawl rate limit is the ceiling the engine respects to avoid overloading a slow server. Crawl demand is the appetite driven by the quality and freshness of content. A healthy home page with new, valuable pages pulls more fetching; a server that times out or a site riddled with errors pulls the rate down.

Why it matters

On small sites, budget is a non-issue; every page gets crawled. On very large sites with tens of thousands or millions of URLs, budget becomes finite enough that how you spend it decides whether a priority page is ever fetched at all. Every byte of wasted fetching on junk URLs is budget that cannot reach a page you actually want ranking.

Spending crawl budget well

1. Keep architecture shallow. Put priority pages at most a few clicks from the home page, because depth and few inbound links reduce the likelihood of a deep page being crawled.
2. Link internally from valuable pages. Fresh links on pages that are themselves crawled bring their targets into the crawl path.
3. Keep canonicals clean. Conflicting or duplicated canonicals force the engine to re-solve which URL wins, which consumes extra fetches. A consistent canonical keeps the crawl on the chosen path. See canonical tags explained.
4. Use the sitemap for discovery of new and orphan pages, since it is a directed hint that pages exist. See XML sitemaps.
5. Let the engine render properly by not blocking CSS and JavaScript with robots.txt; modern crawling needs those assets.

Findable and valuable pages

The engine relearns the site over time. Content found through links and sitemaps repeatedly, that returns fast and gains traction, becomes the pages that get re-fetched more often. Content buried in deep, unlinked tiers, or that keeps returning errors, is deprioritised. Treat crawl budget as a symptom of site health: fix errors, speed up responses, expose useful pages through links, and the effective budget improves on its own.

Blocking useless URLs

robots.txt directs the crawl and keeps crawlers off wasteful areas such as admin UIs, search result pages, and parameter generators. A Disallow does save crawl effort on the blocked paths, because the crawler stops re-fetching those known URLs. It does not deindex a page, however: a URL that is linked or seen elsewhere can still enter the index even though it is never fetched. For a fuller control split, see robots.txt: what it can and cannot do and noindex and meta robots explained.

Common budget wasters to contain:

• Faceted or filter URL explosions that produce thousands of near-duplicate paths.
• Session IDs and tracking parameters appended to every page.
• Archived, tagged, or paginated sprawl with little distinct value.
• Pages that respond slowly or time out, which drag the crawl rate limit down.

Reference summary

The crawl itself lives in the wider pipeline; see how crawling and indexing work.