Crawl Budget Isn't Just Google's Problem: How AI Bots Allocate Attention

Three weeks into a container delay, a US-facing storage brand fields one support ticket: will the wheeled under-bed bin fit a 12-inch frame and clear the elevator? The size chart sits halfway down the page. The AI answer the buyer pasted back omitted it. The crawl log shows why: retrieval requests clustered on facets and color-size variants, rarely reaching the hero SKU three clicks deep. Composite example, not a client.

Crawl budget is not a daily page quota; it is a ranking, and on a large catalog it decides whether your hero SKU or your four-thousandth facet URL gets read.

Budget is a ranking, not a quota

Google’s crawl-budget language was written for indexing. A retrieval crawler is picking sources for a live answer, not filing you for later.

Google splits crawl budget into a rate limit and crawl demand — a frame built for indexing. Retrieval differs: the crawler arrives with a question and a deadline.

Slowness costs asymmetrically: a day late to the index is invisible, four seconds late is fatal.

Depth beats breadth: the crawler enters from cited, linked, and asked-about pages, and past three hops a page does not exist.

Duplication is a tax: ten variants and thirty facet combinations of one product eat attention the hero page needed.

Whether a crawler may enter is a different question, covered by the AI crawler map. This piece is about who gets read first.

Four levers, and only one fits a sprint

Speed is the only lever you can move without a redesign, so it goes first. Link structure is design, freshness is discipline.

Lever	How it works	What you change	How you verify
Response time	A live question can’t wait; a slow page misses it	Cache, drop blocking scripts, stabilize TTFB	P75 per template; 5xx rate
Freshness	Price, stock, and review answers favor recent change	Make price and stock update; touch lastmod only on real edits	lastmod against the change log
Link structure	Repeatedly linked pages within three clicks get revisited	Stable paths from categories to hero SKUs; delete orphans	Crawled share by click depth
Scale and duplication	Facets, variants, and pagination split one page into dozens of URLs	Collapse parameters, merge variants, canonicalize	Unique-content share of crawled URLs

Link structure is the most underestimated: hero SKUs sit in one deep category while facets hang off the header nav, so the crawler takes the nearest door.

Sitemap and lastmod buy guessability, not priority

A sitemap does not raise priority; it lowers the cost of guessing. A fake lastmod is worse than none.

Google has said it ignores the sitemap priority field, and sharding does not reorder a crawl. The value is internal: when hero SKUs, long-tail SKUs, and editorial pages sit in separate shards, you and the crawler read the same tier map, and later trade-offs have coordinates. How much any AI crawler reads sitemaps is not publicly documented by every operator — check current docs, not a borrowed claim.

lastmod fails the same way. Refreshing it on every deploy is self-deception: the template changed, the body did not. To a crawler that weighs freshness, “everything is new every day” is no signal.

Fix access before order: behind a WAF or a JavaScript-only render, a complete AI crawl audit comes first.

Five steps to rank the SKUs

Write the tier rule down, or it degrades into whoever argued last. Tier A is not the best sellers; it is products that sell well and that buyers ask about.

Step	Action	Owner	Passes when
1 Tier	Sort pages into A/B/C by revenue contribution and question density, not sessions	Growth, merchandising	A is at most 20% of SKUs; C has a removal list
2 Reach	Put A pages within three clicks of home or a top category, in a separate sitemap shard	Engineering	30 sampled A URLs within three hops
3 Remove	Take facets, internal search, and duplicate variants out of the crawl surface	Engineering	Unique-content share of crawled URLs rises
4 Trust	Ship price and stock in structured data; update lastmod only on real edits	Engineering, content	20 sampled pages match the change log
5 Re-measure	Count crawl coverage by template and citations in the prompt set	Growth	A before-and-after you can explain

Step 3 is where teams overcorrect into sitewide noindex. Keep facets usable for shoppers but outside the crawl surface: collapse the parameters or canonicalize to the category. Crawlable category and filter navigation covers the design; this piece sets priority.

Example: re-ranking a 100,000-SKU catalog

Example — a hypothetical home-storage store, no real brand or figures: 100,000 SKUs, four-fifths color and size variants, plus facets built from size and material. Before: one full sitemap, lastmod refreshed on every deploy, requests crowded on facets and variants, thin coverage on hero SKUs.

Three changes only: split the sitemap into hero, long-tail, and editorial shards; canonicalize facets and drop them from the sitemap; tie lastmod to real price, spec, and copy edits. No template or copy rewrite.

Acceptance is three counts: crawl coverage by template (definition in crawl coverage as a GEO metric), unique-content share of crawled URLs, and whether the same buyer questions now cite the size chart. Orders stay out: citation and trust sit between a fetch and checkout.

Where this breaks

First, marketplace categories. On Amazon or Shopee, AI reads listings, reviews, and fulfillment metrics; your sitemap is not an input. Fix the feed.

Second, a few hundred SKUs. Skip budget engineering: make everything reachable and crawlable, then spend the time on answer units.

Third, an unreachable site — robots mistake, WAF block, JavaScript render failure — or “crawl more” as the goal. Fix access first: rising requests without rising citations are a content problem, not a budget one.

Fourth, budget is a merchandising decision — which products to push, which long tail to abandon. It belongs to the strategy and market lane, not to SEO or engineering alone.

This month, do one thing: export 30 days of verified AI crawler requests, bucket by template — hero SKU, long-tail SKU, facets, variants, editorial — and compute fetched URLs divided by URLs per bucket. Pick the lowest-coverage, highest-value bucket, apply one consolidation (sitemap exit, canonical, or internal links), then re-run the same count.