A Complete AI Crawl Audit: From robots.txt to Server Logs

Thursday morning in Berlin, a small-space furniture brand pushes a new balcony table live on its German storefront. The growth lead types the question a customer would type — a 60 cm balcony, room for two chairs, folds flat — into ChatGPT and gets four brands back. Hers is not one of them. Engineering opens the live robots.txt: three groups, every one Allow. Nobody touched the template. Both are right, and one question still has no owner: did anything read that product page to the end?

A crawl audit is not a review of robots.txt. It is proof of which of five gates — allow, reach, render, parse, index — is closed, and that proof lives outside the config file.

robots.txt passing clears one gate, not the audit

robots.txt is a preference file. RFC 9309 defines its syntax, not its enforcement: ignoring it breaks no lock. It answers whether a crawler may come — not whether anyone returned or could read the page.

The other four fail in unrelated ways: a WAF returns 403 for a client it misreads, a client-rendered storefront returns 200 with no price, a template ships its size chart as an image, a page sits outside the sitemap. From outside, all four look alike — you are missing from the answer, so the symptom cannot name the cause.

The usual wasted sprint: three weeks rewriting above-the-fold copy because “the model does not understand us yet”, while a bot rule on one template returns 403. Who crawls and what they got are separate questions; the AI crawler map is this audit’s input, not its substitute.

Five gates, five kinds of evidence

Gate	How it fails	Evidence	Passes when
Allow	wildcard group catches a product template; a partial ships noindex	the live file, saved and read group by group	retrieval agents inside an allow group on public pages
Reach	403 or 429, geo-block, verification page	in-market IP with the agent’s user agent; status, headers	200, and the body is the product page
Render	200, but price and specs wait for JavaScript	JavaScript off, diffed against the rendered DOM	title, price, variants, stock visible without JavaScript
Parse	specs inside images; structured data contradicting the page	Product and Offer validation, feed fields against the page	entity, price, availability agree in all three
Index	crawlable, rarely revisited; new URLs missing from sitemap	server logs by path; sitemap lastmod	repeat visits to core templates; new URLs land same day

Only the first gate closes with a configuration change; the rest demand outside evidence — a status code, a log line, a rendered DOM. A report without it is a suspicion list. Order is a cost argument: config, then templates and pipeline, then sitemap and links — which is what stops a team rewriting copy to fix a firewall rule.

Three misreads that void the work

Trusting the user agent. The UA is the easiest field to fake. Two checks hold: forward-confirmed reverse DNS and the operator’s published IP ranges. Fail both and it is an unknown crawler: throttle it, never allowlist a string.

Reading 200 as read. A client-rendered storefront answers 200 and fills the body later. Only a no-JavaScript fetch exposes it, as in whether AI can render your store.

Judging recovery from one screenshot. Answers shift by region, account, and hour: fix a prompt set, wait days, rerun it. Keep arrival and citation apart — a log proves a fetch, not a mention — and score both against an AI visibility baseline.

Two weeks, four steps, one page

Step	Action	Owner	Passes when
1 Sample	one market, one template, 20 URLs (new, variant, out-of-stock); 90 days of logs by agent, path, status	Engineering	top five agents and the paths they fetch
2 Gather	four requests per URL: in-market IP, bare agent, JavaScript off, a normal browser; log status, first screen, price	Engineering	a four-column row per URL
3 Attribute	name exactly one gate; if you cannot, the evidence is thin, so collect more	Tech lead	one gate per row, no “possibly”
4 Report	fixes with URLs for engineering, a fixed prompt set for growth	Growth	in two weeks, what changed or why nothing did

Step 3 is where audits die: five suspected causes per template produce a wish list, and engineering fixes the cheapest one, not the closed gate. Attribute one gate, or leave the row blank.

The report stays on one page: template, failing gate, evidence, owner, retest date. No “likely”. A blank row is the next evidence task.

Example: three faults stacked on one template

Example — a hypothetical DTC seller of portable power stations shipping into the US; no real brand, no invented figures. The audit found three gates closed at once: robots.txt allowed everything; a WAF headless-browser rule returned 403 to a retrieval agent on the product template; price was injected client-side, so a no-JavaScript fetch showed a loading state. Fix order was the WAF rule, then a server-rendered price, then sitemap entries. Retested with the same ten questions, the domain began appearing in answers.

The order is the lesson: the WAF change took an hour and made the other two measurable; a server-rendered price first would have changed nothing.

Where this breaks

Marketplaces get little from this. On Amazon or TikTok Shop, AI reads listings, reviews, and fulfillment; fix the feed first. A brand that refuses training restates the goal as “retrieval allowed, training denied” and reports the two separately. Under roughly fifty SKUs the index gate barely exists: run the no-JavaScript and WAF checks and skip the dashboard. If crawling already slows the origin, add rate limits and caching — an audit does not reduce load.

Time is the last limit. Rules, templates, CDN settings, and front-end frameworks reset a gate on deploy: treat this as a quarterly check or a product-template release gate, not a project with an end date. For why the entry point moved, see how AI search splits e-commerce traffic; the rest of the line is on the technical foundation hub.

This month: one market, one product template, 20 URLs. Fetch them from an in-market IP with a retrieval agent’s user agent, then again with JavaScript off. Record status, first-screen text, and price visibility in one sheet. If those columns are not full, do not schedule the content calendar.