Log file analysis for SEO

Why logs are the ground truth

Server access logs record every request that actually reached your server: timestamp, client IP, method, full URL including the query string, status code, bytes, referrer and user agent. Unlike the tools that model or sample crawler behaviour, logs are the only record of what a crawler really fetched. This makes them the definitive source when a page is crawled far more than expected, a URL space is silently consuming effort, or you suspect bot traffic that Search Console aggregates away.

The practical output is a crawl-behaviour audit: how many robot requests arrived per day, which URL patterns they hit, which returned errors, and what share of that activity landed on pages you actually want indexed. That combination is invisible in the Search Console crawl stats alone, which reports the same crawl at an aggregated level.

Collect a useful window

• Gather 30 to 90 days of access logs. A few days hides weekly patterns; a full budget question needs a month at least.
• The combined log format (Apache/Nginx) and the W3C Extended format (IIS) carry the fields you need. Keep host, URL with query, status, user agent and IP.
• If logs rotate faster than the window you want, ship them to local storage or a log tool before they are overwritten.
• If a CDN fronts the origin, keep your cache-vs-origin field so you can tell edge hits from actual origin fetches.

Verify bots by IP, not just user agent

User-agent strings are trivially spoofed, so never trust the string alone. For every crawler you care about, confirm the IP range against the owner's published list before counting it as real crawler traffic:

• Googlebot: verify via the WHOIS/PTR record check on the IP, or Google's published ranges.
• Bingbot, Yandex, Large/ChatGPT-User for OpenAI crawlers: verify against each provider's posted lists.

Most importantly, split bots by purpose. Modern logs separate Googlebot (indexing) from AI and retrieval crawlers (for example GPTBot, ClaudeBot), which have different retrieval goals and may need separate robots.txt handling. Treating them as one "bot" bucket hides a very different fan-out cost.

Find crawl waste

Once requests are filtered to verified crawlers, group by URL pattern and status:

• Parameter and faceted URLs: thousands of ?colour=red&size=large combinations consuming budget for pages you do not index.
• Internal search results and session IDs: crawlable low-value URLs that should be canonicalised, noindexed, or blocked.
• Redirect chains: a rising share of 301s means the crawler burns requests chasing redirects; fix the leak, not the symptom (see the redirect map discipline).
• 404 and 410 responses being fetched regularly: dead URL sources you have not cleaned up.
• 5xx responses: your origin buckling under consecutive fetches (see load balancer 503).

Cross-reference with Search Console

Pair the log numbers with the Search Console crawl stats and the indexing analysis. The classic signal: a URL Googlebot fetched dozens of times in 30 days that is still not indexed points to either a quality problem or a crawl-efficiency problem, and the logs tell you which by showing how the requests are spent. Because crawl budget is rarely the binding constraint under roughly 10,000 pages, focus log-file work where it pays: big sites, faceted inventory, and places where you suspect the crawl has drifted from what you want indexed.