GEO-024·Issue 02·Measurement & Growth·24 / 132
Crawl Coverage: The Forgotten GEO Metric
Crawl coverage is the first leak in the GEO funnel. Define it by template, verify retrieval crawlers in logs, and interpret flat citation growth.
Issue focusGet Seen First: Crawling, Rendering, and AI Bot Accessibility
In the first week of February, a DTC seller of portable espresso makers — Shenzhen operations, US and German storefronts — reviews why ChatGPT keeps naming its two 2023 models. Before Black Friday the team added six colorways and two bundles, each server-rendered and linked from the collection grid. Engineering checks robots.txt, the sitemap, and a few status codes: all open, all 200. Nobody can answer the question that matters — of those eight new product URLs, how many did a retrieval crawler ever fetch?
Crawl coverage is the first leak in the GEO funnel: the share of pages you want AI to read that a verified retrieval crawler actually fetched in a fixed window — and without it, every later optimization is a bet.
Why the first leak stays unmeasured
No assistant platform publishes a per-site crawl report, and visibility tools measure the answer side, after retrieval. The only proof a request arrived sits in server or CDN logs, which teams open after a 500, not monthly.
Three properties keep the number invisible. The denominator needs human judgment; no tool decides which URLs deserve to be read. Crawling is low-frequency: a long-tail category page may be revisited every few weeks, so a weekly rate swings like noise. And coverage produces no revenue, only a ceiling, fixed weeks before an answer omits you. So the metric has to be assigned, not discovered.
Fix the denominator, then verify the numerator
Both halves are negotiable, so define them first.
Denominator: the URLs you want AI to read, grouped by template, canonical only. Product, category, comparison, and editorial pages count; filter parameters, internal search results, cart, and account pages do not.
Numerator: URLs a verified retrieval crawler requested at least once inside the window, that returned 2xx and still showed body copy with JavaScript disabled, deduplicated to canonical.
Three rules follow. A sitewide percentage is useless: faceted URLs drag it into noise, while per-template numbers point at a page type someone can fix. A 200 is not a pass: if the first screen is an empty shell, counting it inflates coverage; sample with JavaScript off. And only verified bots count: user agents are forgeable, so check the source IP against forward-confirmed reverse DNS or the operator’s published ranges. The layers above fetching are covered in a complete AI crawl audit.
Five steps, one table, two weeks
| Step | Action | Owner | Passes when |
|---|---|---|---|
| 1 Scope | Export canonical URLs by template; document exclusions | Growth and engineering | One list per template, reproducible count |
| 2 Wash logs | Pull 90 days, screen by user agent, verify IP, normalize to canonical | Engineering | A fetched-URL set per template |
| 3 Compute | Coverage per template, bucketed as blocked, pending, or stable | Growth | One row per template, plus 20 low-coverage samples |
| 4 Diagnose | On low-coverage URLs only, test shared traits: link depth, entry points, lastmod, sitemap-only | Tech lead | One testable hypothesis per template |
| 5 Retest | Change one template, recompute with the same definition in 30–60 days | Growth | A before-and-after you can explain |
Treat bucket thresholds as an internal convention, not a benchmark: healthy differs by category and template. The signal is the gap between templates, not any single number. Step 4 is where teams go wrong, usually by asking content to do more. Low coverage is usually structural — no crawlable entry, sitemap-only presence, variants forking off the parent SKU — and none of it improves with another article. Entry-point work lives in crawlable category navigation; revisit frequency is a budget question covered by how AI bots allocate crawl budget.
Example: the worst template is rarely the obvious one
Example — a hypothetical DTC seller of cycling accessories with four templates: product pages, category pages, variant pages, and review articles; no real brand, no invented figures. Over 90 days, category pages and review articles look healthy, product pages sit near half, variant pages near zero.
The first instinct is to rewrite product copy. Pull the variant URLs instead: each is reachable only through a dropdown on the parent SKU, has no crawlable link, and is missing from the sitemap. There is no entrance, not a content problem.
The fix is engineering. Give each variant a crawlable URL, list those in the sitemap with maintained lastmod, and fold color and size facts into a spec block on the parent page. On retest that template’s coverage rises while the other three barely move — the evidence you changed the right thing. Citations did not rise in the same window, which is expected: coverage explains whether a page can be read, not whether the model trusts it.
When coverage rises and citations do not
Coverage is necessary, not sufficient. Read three patterns. Still low after the work: suspect the denominator first; templates may be cut too finely, pulling in one-off or soon-to-be-retired URLs, and shrinking it once often beats fixing ten pages. Stable but citations flat: the bottleneck moved, so more engineering wastes the quarter; check answer units and third-party evidence, using a monthly AI visibility baseline to see which layer is stuck. Coverage flat while citations rise: check whether the denominator omitted them; if it did, the metric is already wrong.
Fetching and citation are separated by indexing and re-ranking, so do not read this weekly. A monthly cadence, a frozen definition, and the same URL set are the only discipline; the rest of the dashboard belongs on the measurement and growth hub.
Where this breaks
A few hundred URLs: sample by hand; a report costs more than it returns. When revenue closes inside Amazon or a marketplace, there are no logs to count — use feed completeness and listing-update latency, not the owned-site definition. If you deliberately block retrieval crawlers, low coverage is the policy working; document it before calling it a problem. Skip the exercise around peak season, when launches make the denominator lurch. And keep it out of a KPI: the cheapest way to hit a target is to stuff the denominator with easy-to-crawl pages. This is a diagnostic, not a scoreboard.
Do one thing this month, pick the product template with the most new SKUs this quarter, export its canonical URLs, pull 90 days of logs, and compute coverage for that template. Split the uncrawled URLs into internal link depth and sitemap status, take ten from each, and recompute on the same date next month with the same definition.