GEO-023·Issue 02·Measurement & Growth·23 / 132
Use Logs to Prove AI Crawlers Actually Arrive
Vendor crawl reports cover only their own bots. Server logs prove which AI crawlers reached which product pages, on what path, and with what status code.
Issue focusGet Seen First: Crawling, Rendering, and AI Bot Accessibility
In February, a DTC seller of portable power stations and folding solar panels shipping into the US and Germany brings a good story to the monthly review: chat-assistant sessions have climbed for two months. The engineer pulls the CDN log and tells a different one. Retrieval-crawler requests to product pages stopped the day the store moved to a new CDN, and every 403 lands under /products/. Sessions kept rising because older pages were still cited; the January launches had never been read. Composite example, not a client.
Vendor reports tell you roughly where traffic came from; only the origin log proves who arrived, which page they fetched, and what they got. Without that layer, every crawl problem is a guess.
Reports measure trends; logs build evidence
Google’s Crawl Stats covers Googlebot; Bing’s covers bingbot. Neither sees the retrieval crawlers behind ChatGPT, Perplexity, or Claude — the visitors this issue is about.
CDN and WAF dashboards do classify bots, with three caveats: proportional sampling, short retention on lower plans, and an “unverified bot” bucket that absorbs the rest. Client-side analytics cannot close the gap, because crawlers do not run JavaScript. A buyer who pastes a product URL into a chat assistant triggers a fetch that exists in the access log and in no crawl report.
Split the jobs: reports answer should we worry, logs which layer broke. Crawl failure stacks in five layers — permission, arrival, rendering, parsing, indexing — and the complete AI crawl audit walks them in order. The log is the only artifact with evidence for all five: each layer ends in a request and a status code.
Six fields are enough
An access log holds dozens of columns; crawl forensics needs six, agreed before anyone opens a BI tool.
| Field | Answers | Abnormal |
|---|---|---|
| Timestamp, with zone | Hour of the fetch | Zone unset; night silence read as daytime outage |
| Source IP | Whether the sender is genuine | Reverse lookup misses published ranges |
| Full user-agent | Crawler and version | Two product names; no version |
| Path and query | Content or parameter page | One SKU via dozens of parameter sets |
| Status code | Whether content returned | 403 or 429 on a product page |
| Bytes and duration | Whether content was a shell | 200, bytes far below template median |
Two rows get misread. A 200 does not mean content arrived: out-of-stock templates, client-side fallbacks, and error pages that render only a header return 200, so compare bytes with the template median and treat a thin response as a miss. Duration is not just performance: as response time climbs, crawlers throttle, volume falls, and the graph looks like a block when the origin is slow.
Build a crawl heat map by path template and status
Raw URLs produce an unreadable table — hundreds of thousands of rows once facets and tracking parameters arrive. Roll up to a path template (/products/{sku}, /collections/{handle}, /filter/*) and three cuts become legible.
Between bots. Keep training and retrieval crawlers separate. If training dominates while retrieval sits near zero, the problem is accessibility, not content quality.
Between path templates. Count content pages — product, category, comparison, editorial — apart from parameter pages such as filters, sort orders, and site search. Parameter pages eating most requests is evidence that crawl attention is wasted — the mechanism in how AI bots allocate crawl budget.
Between status codes. The expensive number is not 5xx but 403 and 429 on product pages, plus the “200 but small” set that looks healthy in summaries.
The three cuts stack into a finding you can assign: “retrieval crawlers have not touched a product page in three weeks; requests sit in /filter.” “AI crawling is down” is a headline, not a finding. When the parameter pages are the cause, the fix lives in crawlable category and facet navigation.
Put the crawl layer on the monthly dashboard
A log nobody opens stops being exported three months later. Four steps, no more.
| Step | What to do | Owner | Done when |
|---|---|---|---|
| 1 Define | Which agents count as AI crawlers, time zone, sampling, retention | Growth and engineering | One page; requests outside it never reach the dashboard |
| 2 Aggregate | Daily table: date × bot × path template × status | Engineering | Any day drills down to template |
| 3 Alert | Product-page 200s at zero for seven days; 403 share jumps; response time doubles | On-call | Alert names a path, not “crawl anomaly” |
| 4 Review | Crawl and citation layers, one page, monthly | Growth | You know which pages were fetched but never cited |
Step 1 is the one teams skip: three people compute three numbers, then argue about whose is right. The metric is not total requests but 200 responses on content pages, by count and share.
Example: three changes and one acceptance test
Example — a hypothetical cross-border home-goods store with several thousand SKUs, a faceted catalog, and a live stock endpoint; logs already on the dashboard.
Two signals: parameter pages absorb most requests, and 403s under /products/ cluster on the newest SKUs.
Three changes. The WAF allows verified bot lists and rate-limits only /api/. robots.txt disallows /filter and site search. The sitemap keeps canonical product and category URLs.
Acceptance is distribution, not total volume: 200s on product pages recover, 403s remain only on intended paths, parameter-page requests fall. All three pass before anyone re-runs the prompt set.
Where this breaks
Marketplace-first sellers. Amazon, Shopee, and TikTok Shop do not hand over access logs. Work from platform reports and brand analytics; forcing it here invents numbers.
Heavily sampled or short-retention logs. Sampling follows your plan. These logs support trends and path distribution, not precise month-over-month comparisons; one day’s wobble may be noise.
Questions about citation. A log proves a fetch, never a use. Test quotation against a frozen prompt set, as described in the AI visibility baseline.
Personal-data exposure. Logs carry IP addresses, which are personal data in the EU; put retention, access control, and masking in the definition document before a debugging export becomes a compliance incident.
No engineering time. Start with the CDN’s bot report and answer one question: did a retrieval crawler get a 200 on a product page? If not, the tooling can wait. The measurement and growth hub collects the rest of this line.
This month, export the last complete month of logs, aggregate by bot, path template, and status, then put two lines atop the dashboard: the week with the fewest 200 responses on product pages, and the path where 403s concentrate. Next month the same two lines get a comparison.