robots.txt: what it can and cannot do

What robots.txt is

robots.txt is a plain text file served at the domain root that polite crawlers read before fetching pages. It tells compliant bots which parts of the site they are allowed to crawl. It is a crawl-control convention, not a security mechanism and not an indexing directive, and that distinction decides most of its real behavior.

A typical file looks like this:

User-agent: *
Allow: /public/
Disallow: /admin/
Disallow: /*.pdf$
Sitemap: https://example.com/sitemap.xml

Syntax

• User-agent: names the bot the following rules apply to. The wildcard * applies to all bots unless a more specific group exists below it.
• Allow: and Disallow: list URL prefixes. Disallow: /admin/ blocks any path starting with /admin/.
• The order of Allow and Disallow matters only on the longest-match rule: a longer, more specific match beats a shorter one, so Allow: /public/ can unblock a path that a broader / Disallow covers.
• Wildcard * matches any zero or more characters, and $ marks the end of a URL, so /private$ matches /private but not /private-page.
• Sitemap: is a non-directive line that points bots at your sitemap URL. It is a hint and is not part of the crawl-permission rules.

A common packed example:

User-agent: *
Disallow: /checkout/
Disallow: /cart
Allow: /checkout/confirmation/
Sitemap: https://example.com/sitemap.xml

What robots.txt controls

1. How budget is spent: blocking low-value paths such as search result pages or archives can keep public resource budget on pages that matter.
2. Which heavy or sensitive paths are not fetched, for example administrative areas, staging files, or param-heavy generators.

What robots.txt cannot do

It cannot inject noindex. There is no noindex keyword in robots.txt, and a Disallow does not reliably remove a page from the index. Notably, if a disallowed URL is referenced elsewhere, from a sitemap, an internal link, or the crawl of an allowed page, search engines may still discover and index the page even though they were stopped from fetching it. Because the bot cannot fetch the page, it also cannot read a meta robots tag inside it that would clear indexing.

To remove a page from the index, use noindex via the <meta name="robots" content="noindex"> tag or an X-Robots-Tag HTTP header, both of which require the page to be fetchable. See noindex and meta robots explained.

For an error-level parallel, a blocked or forbidden area is not the same as HTTP 403; that status is a server response, while robots.txt is a bots-only instruction. Compare with HTTP 403 forbidden.

Crawl budget

Every crawler has limits on how many URLs it will fetch in a time window, known as crawl budget. On very large sites robots.txt is one of the levers that decides whether that budget reaches priority pages. Blocking junk paths or huge parameter-generating areas spares budget for real content. On small sites the effect is usually negligible.

Testing in Search Console

1. Open the Search Console URL Inspection tool and enter the URL you want to test.
2. Run the robots.txt analysis inside URL Inspection; it tests the live file against the URL you entered.
3. Confirm the declared behavior: blocked paths show as blocked, allowed paths show as allowed.
4. Check the lines that use wildcards and $ for unexpected matches.
5. Submit the file again after edits, and watch for accidental blocks of CSS, JS, or the sitemap.

Prevention

• Keep the stagger of specific User-agent groups above the general * group.
• Never block CSS or JavaScript, because modern crawl-rendering needs them.
• Where both exist, prefer robots.txt for crawl control and noindex for indexing control, and know which one you intend.
• Retest after every deploy; a small typo can hide an entire section from bots.

Structured sitemaps sit alongside crawling; see the XML sitemap guide and how crawling feeds indexing in how crawling and indexing work.