The AI Crawler Map: GPTBot, ClaudeBot, PerplexityBot, and Google-Extended

Friday afternoon, a US-facing DTC seller of cat litter boxes sees a colleague paste a Perplexity answer into Slack: four brands for a two-cat household, none of them theirs. The product pages did not change. On Tuesday, robots.txt did — rewritten after a scraping spike to User-agent: * with Disallow: /, plus one exception for Googlebot. Google rankings held; the AI answer moved on within a week.

A training crawler takes a copy once while a retrieval crawler arrives for a real question every time — and what robots.txt allows decides whether you appear in the answer, not just the index.

Two kinds of visitor, one user-agent string

One crawler harvests pages into a training corpus and is finished with you. Another fetches sources at question time, and each fetch is a citation opportunity. The economics point opposite ways, yet one Disallow hits both.

User agent	Role	Cost of blocking
GPTBot (OpenAI)	Training corpus	Training gap; ChatGPT retrieval is separate
OAI-SearchBot (OpenAI)	Index for search answers	Fewer ChatGPT citations
ChatGPT-User (OpenAI)	Fetches a page on request	Pasted links return nothing
ClaudeBot (Anthropic)	Training corpus	Absent from training
PerplexityBot (Perplexity)	Indexes answer sources	No Perplexity citations
Googlebot (Google)	Search indexing and ranking	You leave Google Search
Google-Extended (Google)	Gemini training and grounding	Search fine; Gemini cannot use you

Google-Extended is the most misread row: it controls use, not crawling. Block it and pages stay in Search, but Gemini stops training on and grounding answers in them. The roster keeps moving; as of January 2026, cite each operator’s documentation, not a borrowed list.

robots.txt is a preference file, not access control

RFC 9309 defines syntax, not enforcement: compliance is voluntary, which makes robots.txt a statement of intent and a poor firewall.

Matching is per group, and groups do not merge: the most specific group wins, and * adds nothing. So “disallow everything, then allow Googlebot” leaves every unnamed AI crawler fully blocked — the retrieval agents in the opening scene. Rules bind only future fetches.

Intent	Can robots.txt deliver it	Use instead
Recall content already collected	No	Nothing; only future fetches
Keep fast crawling from slowing the origin	No	Rate limits, caching, a CDN
Remove a page from AI answers	Not reliably	Remove or gate it

Access is not comprehension: specs rendered only in JavaScript or inside images still return an empty answer — see whether AI can render your store.

Allow along two axes: who crawls, and what they reach

A blanket ban takes retrieval crawlers down with the scrapers; a blanket allow lets faceted URLs and stock endpoints get polled until the origin slows.

Page type	Training crawler	Retrieval crawler
Product, category, comparison, editorial	Per brand policy	Allow
Internal search and filter URLs	Deny	Deny
Price and stock endpoints	Deny	Deny; serve a feed
Account, cart, admin	Deny	Deny
Spec sheets and size charts	Allow	Allow, if parseable

Blocking product and editorial pages is self-inflicted invisibility; query-string URLs are unbounded crawl space; anything behind a login has no citation value.

A five-step playbook that fits one sprint

Step	Action	Owner	Passes when
1 Inventory	Export 90 days of bot requests from CDN logs; aggregate by agent, path, status	Engineering	Top ten agents and paths
2 Classify	Read operator docs; label training, retrieval, other; pick allow, throttle, deny	Growth and engineering	A written reason per agent
3 Write rules	Apply the two-axis table without letting * catch product pages	Engineering	No collateral damage
4 Verify identity	Forward-confirmed reverse DNS plus published IP ranges	Engineering	20 log lines match
5 Measure	Re-run the prompt set; check where 403s land	Growth	An explainable before-and-after

Step 2 is the one teams skip. Copied blocklists were written for other catalogs in other markets; your logs are the only evidence of who arrived and what the origin can carry.

Step 4 matters: anyone can put GPTBot in a user-agent string, but the source IP is harder to fake. Reverse-resolve the address, forward-resolve it back to the same hostname, and compare it with the operator’s published range file. If both checks fail, throttle or deny the request.

Example: rewriting a robots.txt in four groups

Example — a hypothetical cross-border pet-supply store with a few hundred SKU pages, filter URLs, and a live stock endpoint; no real brand, no figures. Its robots.txt read Disallow: / under *, then a Googlebot group allowing everything.

The rewrite has four groups. Retrieval agents are allowed on product, category, and review pages. Training agents may read public content but not the spec-sheet directory. Every crawler is denied /search, /filter, /cart, and /api/. The stock endpoint becomes signed, time-limited partner links, with anonymous bursts throttled at the WAF.

Acceptance is not orders: retrieval requests to product pages recover, 403s land only on intended paths, and the same prompt set cites your domain.

Where this breaks

First, categories that convert inside marketplaces. On Amazon or Shopee, AI reads listings, reviews, and fulfillment metrics; robots.txt is nearly irrelevant. Fix the feed first.

Second, a brand that refuses training. Licensed material can justify blocking training crawlers, but accept absence in assistants built on trained weights, and record the policy.

Third, active scraping damage or no log access. Slow origins need WAF rules, rate limits, and caching before robots.txt matters. Without logs, skip elaborate rules: confirm nothing important is blocked, then curl core pages with each operator’s user agent.

Fourth, permission is not citation. Crawl access decides entry only; answer units, entities, and third-party evidence are separate work. The four AI search entry points explain why this layer comes first; an AI visibility baseline shows whether it worked, and the technical foundation hub collects the rest of this line.

This month, do one thing: export 30 days of requests with “bot” in the user agent, rank the top five by volume, read their operators’ documentation, then decide allow, throttle, or deny — and leave the reason beside the rule.