GEO-110·Issue 10·Measurement & Growth·110 / 132
Incrementality Experiments: How to A/B Test GEO
A clean GEO split test does not exist: the model owns the assignment. Four designs that survive, a sample-size rule, and a one-page verdict you write first.
Issue focusMeasurable: AI Visibility, Attribution, and Experiments
October budget review at a DTC brand selling espresso machines into the US and Germany. AI-referred orders have climbed for three months. The CFO asks the only question that matters: how many of those orders would we have lost without the GEO work? Nobody can say — that quarter the team also rewrote sixty product pages, swapped the comparison table, and doubled paid social.
You cannot run a clean on-site A/B test on GEO — the model owns the assignment, exposure happens before your site, and the sample is thin — so you prove incrementality with market splits, time series, holdouts, and resampling, and write the counterfactual down before you begin.
Why the assignment is not yours to control
A web A/B test assumes one URL serves two versions to randomized groups. GEO breaks that at four points.
One index, one version. The assistant crawls the live page; you cannot ask one index to hold old and new copy, or route half of retrieval to last month’s table.
Exposure happens upstream. The buyer reads the answer before arriving; readers who never click leave no visit to count, let alone a control group — the gap behind AI traffic attribution.
The sample is thin. A mid-sized market may see a few dozen AI-referred orders a month; one promotion outweighs it.
The effect spills. Better comparison pages return as branded search and repeat purchase weeks later, mostly as Direct. Blending the four entry points averages four different tests.
Four designs, chosen by what you can randomize
Start with the unit you can assign: market, time window, or page group. The last row is a discipline, not a design.
| Design | What gets assigned | Question it answers | What it costs |
|---|---|---|---|
| Market split | A country or region | Extra orders this batch produced | Markets differ; pair them first |
| Time series | A launch date and a control series | Whether a lift followed launch | Promotions and season contaminate it |
| Holdout | A page group or category | Are the untouched pages worse | You must leave them alone |
| Resampling | The same prompt, asked again | Signal or noise | Cuts error; proves no causality |
A market split holds only if the curves ran parallel for eight to twelve weeks; otherwise the pairing fails. A time series with no control market overstates results: peak season and paid spend sit in the same weeks.
A holdout is cheapest: cut a category out of the program, keep it untouched, and you own a permanent counterfactual — if you can watch it lag without intervening.
Resampling proves nothing alone, but without it the other three mislead: ask the same question on two days and the wording, shortlist, and sources all move. One snapshot is a coin flip.
Size the test before you run it
Write down two numbers: baseline weekly AI-referred orders, and the smallest change you care about.
Take a market at thirty orders a week: a hundred and twenty over four weeks. That volume can show a quarter swing, not five percent; if the decision hinges on five percent, the window is too short or the unit too small.
The primary metric is orders or contribution margin, never sessions; sessions rising while orders stay flat is the most common false win. Process metrics — citation share, shortlist inclusion, answer correctness — are leading signals, defined in the four layers of AI visibility, but they cannot replace the revenue test. Do not add sample until the number looks better: fix the verdict line first, read it at the end.
Five steps from a claim to a verdict
| Step | Action | Owner | Passes when |
|---|---|---|---|
| 1 Write the counterfactual | Absent this change: which market, which weeks, how many orders | Growth lead | Finance can repeat it unchanged |
| 2 Pair or hold out | Two similar markets, or a category left out | Growth, engineering | Baselines parallel for eight weeks |
| 3 Freeze variables | Price, promotion, spend, launches stay put | Operations | The freeze is in the weekly notes |
| 4 Set the verdict line | Metric, detectable effect, window, stop rules | Growth, finance | Both sides sign before launch |
| 5 Interpret and decide | Direction, magnitude, process metrics, retest plan agree | Growth | The report says continue, stop, or wait |
Steps 2 and 3 decay first. Sales wants a launch, paid wants budget, operations wants the promotion; enough reasonable exceptions turn the test into a logbook. Shorten the window before you break the freeze.
Example: a two-market holdout
Example — the brand, markets, and figures are invented.
A team selling robot mops into the UK and the Netherlands picks two comparable markets. Market A rebuilds forty product and comparison pages in one release; Market B stays untouched, with the same budget and promotion calendar, for twelve weeks.
At week eight, AI-referred orders in A moved from eighteen a week to twenty-seven; B moved from sixteen to nineteen. A’s shortlist inclusion rose from three of twenty-four tracked prompts to seven; B did not move. Direction agrees, but after subtracting B’s drift the net is six orders a week, short of the eight set before launch, and the interval is wide.
Verdict: direction supported, magnitude unconfirmed. The team extends eight weeks with B frozen, no added spend, and instruments A’s three most-cited pages to narrow the next round. Had it read “up fifty percent” and rolled out across Europe, B’s three weekly orders of natural growth would have been booked as winnings.
How those pages were rewritten is an answer unit problem; the experiment only decides whether the batch is worth copying.
Where this breaks
Too few orders. At single-digit monthly orders per market, any statistic describes noise; use process metrics plus qualitative evidence, and label it non-causal.
In-platform AI. The marketplace offers no assignment; record listing variables and the before-and-after curve, and discount the conclusion.
Peak season. When demand is jumping, the experiment gets swallowed; avoid the window, or observe without a verdict.
Bundled changes. Rebuilding forty pages answers only “did the batch work”; stage the release to isolate a page, at a longer cycle.
Causality as a dissertation. Leadership needs continue, stop, or scale, not significance; consistent direction and clean process metrics usually suffice.
Privacy shortcuts. Do not fingerprint visitors or stitch cross-site IDs to complete the assignment; that turns an experiment into a compliance incident.
This month, hold one mid-sized market still while the changes land in a second. Before launch, put the primary metric, detectable effect, and verdict line on one page. Eight weeks later, chart both curves and read the sentence you wrote first.
The rest of the line sits on the measurement and growth hub.