Advanced robots.txt directives

The full grammar

The robots.txt basics cover User-agent, Allow, Disallow and the general shape. The advanced surface adds three things: wildcards, the dollar anchor, and the Sitemap line being a separate, non-rule field.

Wildcards and the dollar

User-agent: *
Disallow: /private/*.pdf$
Allow: /private/guide$

• * matches any number of characters (including none): /private/* matches /private/, /private/a, and /private/deep/file.
• $ anchors the match to the end of the URL: /private/*.pdf$ matches /private/foo.pdf but not /private/foo.pdf?download=1 (the URL still contains ?download=1).
• A long pattern matches before the wildcards for the peer comparison: /private/ beats /private/* when both would apply.

The Sitemap: line

Sitemap: is not a rule. It is a hint to crawlers about where the XML sitemap lives, and it is not subject to User-agent groups (search engines ignore a Sitemap: inside a group aimed at a different bot). The exact text from the engines: the sitemap field is not tied to any specific user agent. The value must be a fully qualified absolute URL (https://example.com/sitemap.xml). It is case-sensitive in the URL part, and you can repeat it (one per sitemap).

Longest-match and matching

Google's guide describes the "longest match" rule precisely: a matcher that includes a path up to the first wildcard, or a "word boundary" (the $ or the end-of-the-URL), whichever comes first. A rule such as Disallow: /forum/* matches any URL starting /forum/. The classic subtlety is that a match is a prefix match of the URL path, not of the whole file, so Disallow: /private matches /private, /private/ and /private-key (the reverse is not implied), and Allow: /private/ does not allow /private-key.

Group matching and precedence

Rules apply to the most specific (longest) matching rule for the URL among those in a matching user-agent group. Order in the file matters only when two rules within the same group match the same length; otherwise the longer (more specific) route wins, so a /public Allow can override the broader / Disallow. Groups are independent. English defaults: no rules means everything is allowed; a Disallow: with an empty value means allow all within that group.

The key differences between the group-level and URL-level reality:

• Allow is a crawl permit, not an index permit. An Allow does not make the URL indexable; it only lets the crawler fetch it.
• A non-matching group (e.g. a User-agent: googlebot-news without a tail rule) defaults to allowed.

The indexing boundary (meta robots vs x-robots vs robots.txt)

The three controls form the complete indexing story:

• robots.txt is crawl control only. It stops a fetch, cannot express noindex, and does not reliably deindex a URL referenced elsewhere.
• <meta name="robots" content="noindex"> is index control inside the page HTML. It requires the page to be fetched.
• X-Robots-Tag (via the server or CDN) is index control on the HTTP response, and the only one that works for non-HTML files (PDFs) and for responses the page head cannot express. The header guide has the full directive set.

The one rule that confuses everyone: robots.txt can block, but cannot cause a page to be *removed*. Google's model: if a URL can be discovered while robots.txt disallows crawling it, the URL can stay in the index with limited data. To remove, use the HTML or header noindex.

When advanced robots.txt matters

• Filter parameter traffic where * and $ compress many URLs into one rule instead of hundreds of exact lines.
• Keeping the Sitemap hint global without gating it to a specific bot.
• Expressing "crawl most but not /*.pdf$" type shapes.

Clarity beats cleverness: a single Sitemap: line at the footer, a few broad Disallows, and understanding that the meta/X header owns indexing will outperform a "clever" 200-line robots.txt with 40 wildcard combos. The noindex meta article is the pairing that turns robots.txt into a complete deindexing policy.