Cloudflare, Akamai, and WAFs: Don't Lock AI Crawlers Out

Two weeks before Black Friday, a US DTC seller of portable power stations sees its origin time out under a scraper wave. Ops adds three WAF rules: block requests without a browser user agent, 429 above a per-IP rate, 403 anything with a low bot score. The site recovers by evening. A week later, ChatGPT no longer names the brand for a camping power station, Perplexity cites two competitors, and Google organic has barely moved.

A WAF is not neutral: its defaults decide which pages an AI can read, and retrieval crawlers are the first casualty of rules written for scrapers.

The firewall is enforcing a content policy

A firewall assumes a human is reading the page; anything that does not look human is suspect. A retrieval crawler looks nothing like one: no cookies, no JavaScript, no clicks, never finishes the challenge script, and refetches the same URLs in a burst. Bot management scores each as a deduction.

The chain is short: a suspicious score, a JavaScript challenge the crawler cannot execute, a 403 or an interstitial with no product copy. The crawler records a failure, lowers its rate for that path class, and may not return. The dashboard says “AI stopped citing us”; the log says “we refused it.”

The layer is easy to miss: it is not in robots.txt. robots.txt is a declaration; the WAF is enforcement, and the two can openly contradict each other. Diagnosis starts at edge logs, as in a complete AI crawl audit.

Four ways bot management catches the wrong visitor

They stack, and vendor lists change, so read operator documentation, not borrowed lists.

Control	What it appears to do	Effect on a retrieval crawler
Verified-bot allowlist	Admits only crawlers passing IP and user-agent checks	Skews to search engines and large platforms; anything missing is blocked by default
JS challenge or CAPTCHA	Separates people from scripts	The crawler cannot execute it and receives a page with no body
User-agent keyword blocking	Stops impersonating scrapers	Catches crawlers that announce themselves; a scraper edits one header and gets in
Per-IP rate limiting	Protects the origin	Retrieval agents fetch at high rates from few addresses and hit the limit first

The first row is the one teams misread. “We never blocked AI” is compatible with a deny-by-default allowlist: not innocent until proven guilty, but unknown until approved. The cost is not a click; it is the right to be cited. The third row punishes crawlers that follow the rules while a scraper edits one header and walks past — the argument in whether to allow or block AI crawlers in robots.txt, repeated because UA blocking keeps returning as “security hardening.”

Allow in four steps, in this order

Order matters: throttling before verification treats every crawler as one blob, and rules you cannot defend do not survive an incident.

Step	Action	Owner	Passes when
1 Evidence	Export 30 days of edge logs hitting bot rules or returning 403/429; group by user agent, IP range, path	Engineering	Ten most-blocked agents and their paths
2 Verify	Reverse-resolve, forward-resolve, compare with published operator ranges	Engineering	20 sampled lines resolve both ways inside the official range
3 Group	Label retrieval, training, unknown; allow product, category, and editorial pages to retrieval	Growth and engineering	A reviewable reason beside every rule
4 Throttle	Limit parameter, search, and API paths only; 429 plus Retry-After, never 403	Engineering	403s land only on intended paths

Step 4 is where most teams go wrong. In HTTP semantics, 403 says the request will not be fulfilled however often it is retried; 429 says the client is too fast and may return. A protocol-respecting crawler backs off on 429 and gives up on 403. Only one preserves the relationship.

Example: a fix that changed three things

Example — a hypothetical EU home-goods store, a few thousand SKUs, no real brand or figures. Before peak season it rate-limited edge traffic on a site-wide path match with a 403. Product and filter pages were caught together, retrieval requests to product URLs fell to zero within a week, and citations went with them.

No vendor changed. Three rules did: 403 became 429 with Retry-After; the path match exempted product and category pages while /search and /filter kept the limit; the verified-bot allow switch went back on, with the reason in a rule comment.

Acceptance was three signals, not revenue: 403s on product paths at zero, 429s confined to parameters, and the prompt set citing the domain again.

Read 403 and 429 separately

Adding 403 and 429 into one “blocked requests” number merges a permission decision with a capacity decision, and they need opposite responses.

Watch three signals: the share of 403s on product and category paths, near zero; the distribution of 429s, clustering on parameters by design and scattering across product pages by accident; and challenge pass rate, near zero for a real crawler since the challenge tests for a browser.

Logs prove a crawler arrived and was refused, not that it would have cited you. Re-run a fixed prompt set to judge the repair, as in using logs to prove AI crawlers arrive. On large catalogs, protect the templates that matter first, per how AI crawlers allocate crawl budget.

Where this breaks

Compliance comes first: paths touching user data, payment, or member content stay closed; they were never going to appear in an answer.

Some origins cannot absorb the traffic: add caching and static rendering first. A store that falls over is more invisible than a throttled one.

Training and retrieval are separate decisions. Declining training on licensed content is legitimate; declining retrieval too deletes you from the candidate set for nothing.

If most revenue closes on Amazon or Shopee, the WAF is not your bottleneck: those engines read listings, reviews, and fulfillment metrics edge rules never touch.

Treat a one-click “block AI crawlers” toggle as a policy choice, not a default: it blocks a whole category, and most stores need part open. The rest of this layer sits on the technical foundation hub.

This month: pull 30 days of edge logs, filter 403s on product paths, rank by agent. For the top three that pass verification, swap 403 for 429 with Retry-After or add them to the verified-bot allow group. Re-run the same prompt set three days later.