GEO-014·Issue 02·Technical Foundation·14 / 132
Allow or Block AI Crawlers in robots.txt?
Blocking AI crawlers often closes the answer door too. Map pages into three layers, write robots.txt by group, and price the cost of a mis-block.
Issue focusGet Seen First: Crawling, Rendering, and AI Bot Accessibility
A DTC brand selling portable power stations to American campers puts three people on one call. Legal found a competitor republishing its installation guide. Growth watched Perplexity answer “best power station for car camping” without the brand for a second week. Engineering fixed the scraping in December with User-agent: * and Disallow: /, plus one Googlebot exception. All three complaints are true, and the call is stuck on one question: who gets in?
Allow or block is not an ethics question but a commercial one: sort pages into three content layers, decide which crawler class may touch each layer, and write the reason above the rule.
Sort pages before you argue about bots
Most teams debate whether AI crawlers should be allowed at all. The answerable question is which pages need which visitor; one file produces opposite outcomes on different URLs.
| Content layer | Typical pages | Who gets in | Why |
|---|---|---|---|
| Traffic assets | Products, categories, comparisons, guides | Retrieval: allow. Training: brand policy | Answer material; blocking hands the question away |
| Transaction data | Prices, stock, member pricing, coupons | Nobody; serve a feed or signed link | Polling slows the origin; prices must match checkout |
| Proprietary assets | Manuals, internal training, unreleased photos | Training: close. Retrieval: case by case | Real gain, real cost: answers explain less |
The table breaks the urge to block everything. The counter-example is routine: User-agent: * with Disallow: / and a Googlebot exception. Rankings hold; retrieval agents stay out because an unnamed agent falls into *. Groups do not merge. Mechanics in the AI crawler map.
Allowance compounds, blocking saves once
A retrieval crawler arrives per question, so every fetch is a chance to be quoted. A training crawler takes one copy and leaves. That asymmetry drives the decision, and teams price it wrong.
- Upside: every question is a chance to be cited. It rarely orders that day, but it enters the buyer’s question chain.
- Cost: bandwidth, copied content, competitors studying you. Caching, rate limits, and WAF rules handle the first two; the third is competition.
- Irreversible cost: collected content cannot be recalled, and a citation slot is easier to lose than to reclaim.
The default that survives a quarterly review: retrieval allowed, training discussed separately, transaction data behind an API, not crawlable URLs. Keep “stop scraping” and “stop being cited” as two decisions.
Write groups by purpose, not by company
The hard part is grouping, not syntax. Three functional groups beat a dozen company-named ones; a new crawler joins a class.
| Group | Members (read vendor docs) | Rule | Purpose |
|---|---|---|---|
| Retrieval, user-triggered fetches | OAI-SearchBot, PerplexityBot, ChatGPT-User, Claude-User | Allow public pages | Citation path stays open |
| Training | GPTBot, ClaudeBot, CCBot | Allow or deny by layer | Optional corpus, negotiable conditions |
| Usage control | Google-Extended | Own group, with comment | Not crawling; Gemini training/grounding |
| Fallback | * | Deny /search, /filter, /cart, /api/ | Crawl space, no citation value |
Google-Extended is misread as “block Google”. Search indexing is untouched; it states a training position, a brand decision, not a traffic one. The * group is a fallback, never a home for public pages. Facets belong on the deny list, and crawlable category navigation exposes the same products without the parameter maze. Names change; as of February 2026, read each operator’s docs, not a copied blocklist.
Five steps inside one sprint
A rule nobody can verify is a guess with a date stamp; each step ends in an artifact.
| Step | Action | Owner | Passes when |
|---|---|---|---|
| 1 List URLs | Ten URLs per layer, answer material marked | Operations | Products, categories, policies, endpoints |
| 2 Decide | Allow, throttle, or deny per agent, one reason | Growth and legal | No reason reads “because everyone blocks them” |
| 3 Write rules | Three groups, a comment above every rule | Engineering | Each comment names what it protects |
| 4 Verify identity | Compare claimed agents with operator IP ranges | Engineering | 403s appear only on intended paths |
| 5 Test | Re-run the prompt set, compare cited sources | Growth | You can explain what changed |
Step 5 usually stops at “requests are in the log”. Crawl volume proves arrival, not use. Logs answer “did it come”; the prompt set answers “did it use us”. Large catalogs should rank URLs first; AI bots allocate attention unevenly.
Example: two lines become four groups
Example — a hypothetical US cycling brand with a few hundred SKU pages, filter URLs, a live stock endpoint, and an installation manual PDF. No real brand, no figures. Its robots.txt read User-agent: * with Disallow: /, plus a Googlebot allow-all.
The rewrite has four groups. Retrieval agents may read product pages, categories, and guides. Training agents may read public pages, not the manual directory. Every crawler is denied /search, /filter, /cart, /api/. Stock moves to signed links that expire, anonymous bursts throttled at the WAF.
Acceptance is not orders: retrieval requests to product pages recover, 403s land only on intended paths, the same prompts cite the domain. Reverting the file is not recovery: crawling has a rhythm, indexes lag, and the slot may already be taken. Freeze one version, re-test for two weeks. Allowing first and narrowing later is cheaper than denying first and reopening; the second order pays after the loss.
Where this breaks
Sales concentrate in marketplaces. On Amazon or Shopee, AI reads listings, reviews, and fulfillment metrics. robots.txt barely matters; fix the feed first.
The content is the product. Paid reports, courses, and exclusive drawings lose revenue when copied wholesale. Closing training, even retrieval, is rational if you accept absence from the answer.
Scraping has destabilized the origin. robots.txt binds only the compliant. WAF rules, rate limits, and caching come first.
Legal owns the decision. Written policy and approval records matter; a file that flips weekly destroys the log baseline.
You are not sure you are mis-blocked. Audit allow, reach, render, parse, and index before setting policy; a complete AI crawl audit covers the sequence. Permission is not citation: access decides entry only. The technical foundation hub collects this line.
This month, do one thing: rewrite robots.txt into the three groups, put one reason comment above every rule, then ask the same question in ChatGPT and Perplexity and record the cited sources. That recording is next month’s comparison set.