GEO-118·Issue 10·Technical Foundation·118 / 132
AI Answer Monitoring Tools: Build, Buy, and Validate
Coverage numbers demo well and validate nothing. Five checks, one veto, a three-table in-house build, and an agreement-rate audit for AI answer monitoring.
Issue focusMeasurable: AI Visibility, Attribution, and Experiments
The vendor demo fills a wall screen: eleven models, thousands of samples a day, a mention rate rising all quarter. A US DTC pet-hydration brand signs a twelve-month license. Three months later its ops lead cannot say which prompt drove last week’s jump. The dashboard holds scores, the backend weekly averages, and the account manager says models fluctuate.
A monitoring tool is worth what you can reproduce, export, and recalculate; coverage is the number a demo can inflate without proving anything.
Eleven models is a demo number, not a buying criterion
An AI answer is not a fixed object. Ask Tuesday and Thursday, in another region, under another account, after a point release, and the text differs. A tool sells a sample estimate, not a measurement — legitimate only if written down: prompt set version, model identifier, region, language, repeats, time window. Lose one and two months of curve are two experiments on one axis.
Vendors lead with coverage because it demos well. Eleven models fill a slide; reproducibility cannot be shown in an hour. The sheet asks for model count, not whether you can pull one record back out: answer, citations, timestamp, model, prompt ID.
Then judgment: does the product-line name count apart from the brand? Does “I would not buy this one — the pump is audible at night” count as a mention? Does a footer link with no matching sentence count as a citation? Three defensible answers, three percentages from one dataset — unpublished rules mean you bought a conclusion, not data.
The pattern is common: mention count in the Monday report. It rose most weeks, so meetings had something to discuss; six months on, the candidate list, the reasons in answers, and revenue were unchanged. Mention is the layer noise lifts easiest — one alias-table edit moves it several points.
Five checks, one veto
Score vendors on what you can verify after signing, not the demo.
| Check | Ask the vendor | Passes when |
|---|---|---|
| Sampling control | Model identifier, region, language, repeats, window, versioned prompt set? | Two people run one config, comparable records |
| Verbatim retention | Full answer, citations, timestamp, model identifier per record? | JSON or CSV export, not screenshots |
| Published rules | How are mention, citation, recommendation decided? Aliases, sub-brands, negations? | Recompute from raw text |
| Validation support | Blind coding, logged disagreements, an agreement rate? | Thirty records reconcile line by line |
| Data and contract | Raw data owner, retention, retrieval after termination? | Export format and deadline in the contract |
One veto: a tool that cannot export the raw answer is not purchased, whatever its coverage; more models on an unreproducible score only make the number larger.
The minimum in-house build: three tables and a scheduler
For one market, one engine, twenty to forty questions, building beats buying: a prompt table (ID, market, language, layer, buyer’s words), a sampling config (identifier, region, language, repeats, window), an answer table (prompt_id, model, sampled_at, region, answer, citations, status), a scheduler, and a folder of raw JSON.
| Step | Action | Owner | Passes when |
|---|---|---|---|
| 1 Fix the prompt set | 20–40 real questions, four layers, versioned; edits start a new baseline | Growth, content | Every prompt has an ID |
| 2 Write the config | Model, region, language, repeats, window in a file | Engineering | A second person reruns it unchanged |
| 3 Store raw first | Answers land untouched; parsing is a separate job | Engineering | Ten records open with text and citations |
| 4 Write the rules | Alias table, negation handling, citation test | Content, engineering | Two coders explain every disagreement |
| 5 Ship a dashboard | Rates plus absolute counts, by engine and market | Growth | The denominator is printed |
Step three gets skipped. Parse before storing and the rules weld into the data; change one and history is void. Keep raw, rerun the parse — the structural advantage of building, and the only way to answer “why did this rise” later. Whether to run the pipeline is the trade-off in build, agency, or tools.
Example: the dashboard said 58%, the humans coded 39%
Example — brand and figures invented; the method is reusable.
Thirty English prompts, three runs every other day for four weeks: 360 answers. The tool reports 58% mention; ops blind-codes sixty and 39% hold. Aliases: brand and product line counted as two entities, so one sentence scored twice. Negation: “not the one I’d buy — the pump is audible at night” matched a keyword and logged positive. Citation: footer sources counted, no body sentence matched the page — the expensive one, since it decides whether the model used your words at all.
Rewritten rules recompute to 41%, two points off the human code. They also moved the denominator from all records to valid answers, dropping errors, refusals, empties; the trend held still. Nothing was replaced. The rules were written down.
Validate the instrument before the brand
Run it backwards and you explain noise as strategy.
| Bias | Looks like | How to test |
|---|---|---|
| Sampling | Same prompt, different answer next week | Rerun one config twice in a day |
| Scoring | Negation logged positive; alias missed | Blind-code thirty, compute agreement |
| Denominator | Rate jumps when errors are excluded | Report counts beside rates |
| Model drift | A model swap breaks the line | Record the identifier; new baseline |
Define agreement before the first sample: two coders label independently, the denominator is the sample, disagreements sorted by type, and the dominant type is the rule to fix. Instrument error deserves the scrutiny brand performance gets — these rates sit inside the four layers of AI visibility.
Where this breaks
Three teams should not buy yet. Monthly question volume in single digits: ratios wobble weekly and a qualitative log beats a license. An unresolved brand entity: three spellings, no off-site confirmation, sampling returns zeros, and the tool amplifies them — settle whether your brand entity exists first. Platform assistants as the main channel: answers shift by account, region, and session, prompts cannot be rerun, and the only comparable inputs are listings, reviews, and fulfillment.
Two further limits. A tool cannot prove causation; lift needs an experiment. Automated querying runs into terms of service — read them before scripting a chat interface. The dashboard says the answer changed, not why, and cannot promise the next sample repeats it.
This month, take twenty real buyer questions, run a tool and a human the same day, export the raw answers, compute one agreement rate. If they disagree on more than one record in five, fix the alias table, the negation rule, and the citation test before the renewal call. The rest of this line is on the technical foundation hub.