Comparison Page A/B Tests: Did AI Actually Change Its Answer?

A DTC dehumidifier brand ships from New Jersey to US buyers and from a Dutch 3PL to Germany; its comparison page leads with a JPEG of three rivals. In May the content lead rebuilds it as an HTML table, adds a “Not for” row, and ships. Three days later someone asks ChatGPT for a dehumidifier for a damp basement flat, under $300, shipped to Germany. The brand appears, page linked. The screenshot lands in Slack; the rebuild is a win. Two weeks later, same question, new account, German: gone.

An AI answer is a draw, not a page: one screenshot proves you were sampled once, and only frozen prompts, repeated draws, one changed variable, and a verdict written in advance separate improvement from luck.

Three sources of noise, one of them yours

The same question from another account, country, or day returns different wording, order, sometimes names; three layers explain it.

Retrieval. The prompt is rewritten before it is searched — capacity, noise, filter cost become sub-questions — and each matches a different slice of the index. Language, region, and freshness decide which pages enter; a US page whose German version lags reads as another brand.

Generation. The same retrieved set can be summarized differently. Some engines build from sources, others lean on model memory, and none guarantees a sentence twice.

Propagation. A new version must be fetched, indexed, then retrieved; “results the day after launch” usually mean the old page was read, or this draw fell your way.

Until a crawl log shows the new version fetched, you are not testing — you are collecting anecdotes.

Three outcomes per answer, one of them the verdict

A redesign moves more than one number, so score three layers separately.

Layer	What you count	What it is for
Citation	A link or a named source	Diagnosing retrieval
Inclusion	Brand in the shortlist, with a condition	Deciding; closest to revenue
Accuracy	Specs, price, and “not for” repeated correctly	Repairing; a wrong number costs more than absence

Inclusion rate is the primary metric: prompts where you are named an option, divided by prompts asked. Citation is diagnostic: a page cited but never recommended was filed as reference material, not a product — which is why an assistant names three to five products and stops.

The recurring counter-example: to look even-handed, a team deletes its “not for” column. Citations climb, answers say “the brand does not state who it suits,” inclusion falls. Screenshot-chasing also trains a team to sand down the difference it was shortlisted for — the one asset a smaller brand has, per the challenger’s opening.

Freeze the questions, then change one thing

A page test splits traffic on your site; here the outcome happens inside someone else’s model, so only half the variables are yours, and those must be fixed.

Step	Do this	Owner	Passes when
1 Write a hypothesis	Which page, which change, which layer	Content, growth	Names page, change, metric
2 Freeze the prompts	20–30 real questions from search, tickets, ads; one language and region	Growth	Each traces to a typed question
3 Sample the baseline	Before the change, each prompt on each engine five times (Example)	Growth	Date, account, region, model logged
4 Confirm exposure	Wait for crawl logs; hold price, title, links	Technical, content	New-version fetches appear
5 Compare in pairs	Same prompt before and after: improved, worsened, unchanged	Growth	You can name the moved prompts

Step four is the one teams skip: testing before the crawler fetches the new table measures the old page and credits the new. If the table is still an image, the rebuild changed a file, not a sentence; machine-readable comparison tables covers that repair, and no A/B test reveals it.

Step five is paired. Three improvements and two regressions out of 24 are noise; twelve improvements and one regression are a change. What to change is separate: comparison page structure.

Write the verdict before the first sample

A verdict written after the data is a rationalization.

A workable rule (Example): 24 prompts, five runs each; pass if four or more improve, no more than one worsens, and the direction holds at a two-week recheck.

Stability beats size: one appearance that fades is a fail. Adding samples until a difference looks significant is not rigor; if one prompt of 24 moved, the change was too small or the prompts missed the question. Skip platform “average rank” too; the method is unpublished.

Example: two weeks, one variable

Example — brand, rivals, and figures invented; only the procedure is reusable.

Baseline: 24 English prompts across three scenes (damp basement, nursery, rented flat), three engines, five runs each, 360 samples. Included in 6 of 24, cited in 4, five spec errors.

One change: JPEG to HTML table, units in the header row, “not provided” for a dash, a “not for” row; price, title, and links untouched.

Recheck: included in 11 of 24, six improved, one worsened; spec errors down to one; citations 4 to 5. Verdict: keep it.

Citations barely moved while inclusion nearly doubled: the table was used as evidence, not source, and a citation-only scorecard would have dropped the change. The regression, “rented flat,” saw answers name a rival with a larger tank — a missing scene page, not a table problem. Build the page next; leave the table alone.

Where this breaks

The sample budget is too small. If AI referrals are too rare to form a distribution, keep the log and skip the verdict.

The answer lives in a marketplace. Platform AI gives no control arm; before-and-after with listing variables recorded is not an A/B test, so discount the conclusion.

You are absent to begin with. Fix the entity and off-site evidence first; a better table has nothing to attach to.

Several things moved at once. Price, promotion, stock, and title changes make attribution impossible — postpone, or log without judging. Strong personalization narrows any claim to direction, not cause.

Comparative claims need substantiation. US and EU rules put the burden on the advertiser: have counsel read the table, and never invent a rival’s weakness to earn a citation.

This line sits in measurement and growth. This month: take the comparison page buyers ask about most, write the hypothesis as one falsifiable sentence, freeze 20 prompts from support tickets, sample five runs on one engine, and check the crawl log before reading an answer. Without a baseline, a redesign cannot be told from a good draw.