GEO-047·Issue 04·Measurement & Growth·47 / 132
Scoring Answer Quality Before Using an LLM as Judge
An LLM judge rewards fluent prose, not correct facts. Five criteria, decidable bands, and a calibration loop that keeps answer-quality scores comparable.
Issue focusAnswer Units: Writing Category and Product Pages for GEO
In April, at a composite DTC countertop-appliance brand selling into the UK and Germany, a chart shows answer quality up six weeks straight, 6.9 to 8.4. Then a support lead pastes a Perplexity answer into Slack: 2,000 W for a 1,500 W air fryer, “European plug” for the UK variant, a warranty retired in January. Four of the five answers it praised that week carry one such error.
An LLM judge can triage a hundred answers down to the ten worth reading, but it cannot produce an absolute score for a bonus plan — so write accuracy, specificity, citation, freshness, and brand fit as decidable bands first.
Why the scores climb on their own
A judge model has none of your master data, policy pages, or test reports. It infers truth from how true the prose sounds: complete-looking structure wins even when empty, adjective density wins because “efficient” and “versatile” co-occur with praise, and confidence wins while a qualified sentence (“at 220–240 V”) reads as hedging.
Then the target gets optimized against. Once a score reaches a weekly report, the team removes deductions instead of errors: “example” deleted, “subject to change” dropped, every uncertain claim made absolute. The number rises as verifiability falls.
Noise compounds it. Example — three runs over one set of 40 answers put the mean between 6.9 and 7.6. Absolute scores do not compare across runs; ranking inside one rubric version does, and so does the disagreement list.
Five criteria, each with a place to check
A criterion with no evidence source gets a tone score. Point each at something a reviewer can open.
| Criterion | Question | Evidence | Typical failure |
|---|---|---|---|
| Accuracy | Matches master data, policy | PIM, policy page, test reports | Wrong wattage or warranty |
| Specificity | Checkable models, values | Specs, compatibility list | “High performance” for a number |
| Citation | Each claim sourced | Specs, reviews, certificates | Self-citation, dead links |
| Freshness | Price, stock, policy current | Update, policy dates | New date, unchanged body |
| Brand fit | Same term per market | Entity table, market glossary | Own wording per site |
The five are not equal. Accuracy is a gate; the other four carry the score. Otherwise a specific, well-cited, fresh paragraph with the wrong capacity scores high — and capacity errors produce returns. Brand fit is not a copy preference: “mini oven” on one storefront and “toaster oven” on another can split one product into two entities. The fix is less translation than whether the brand entity exists.
Decide the verdicts before arguing about the number
The rubric’s value is in its rules, not its scale: what is disqualified, what goes back for material, what clears the bar.
| Verdict | Trigger | Handling |
|---|---|---|
| Reject | Accuracy 0: one fact contradicts policy | Rewrite; out of the mean |
| Return | Citation 0, or expired price or policy | Add sources or update, re-score |
| Pass | Accuracy 2, others 6 of 8 | Into the next sample |
| Template | All five 2, one off-site citation | The writing model |
Score the answer, not the page: the object is question plus verbatim answer plus citations, since one page yields different paragraphs per prompt. Paragraph boundaries decide what gets lifted, so semantic HTML for extraction is a scoring input, not a nicety. Change one variable at a time: rewriting rubric and content together makes attribution impossible.
A counterexample: one team added “mentions the brand’s core benefit” and every answer scored full marks — the phrase is on every page. A criterion that cannot separate good from bad is worse than a wrong weight — delete it.
Calibration, sampling, and versioning
Model grading passes a human exam first: humans agree, then the model agrees with them.
- Human pass. Pull 30–50 verbatim answers AI returned, not your pages. Two editors score independently; if one gives 1 and the other 2 on the same sentence, the bands are wrong, not the prompt.
- Model alignment. Read the disagreement list, not the mean gap, which hides systematic bias. Most judges favor length, structure, and confidence; naming that bias works only when the prompt carries the rubric, band anchors, and source documents, and demands a score, a reason, and an evidence location.
- Sampling. Stratify by question type — category, comparison, constraint, brand — from the frozen baseline. Swap questions and the ruler changes. Sample answers, never pages.
- Versioning. Re-score monthly; when the rubric changes, re-run ten anchor answers, or last month’s 7.2 and this month’s 6.4 mean nothing. Log four fields: rubric version, prompt template, model version, sampling date.
Example: one rubric, rebuilt
Example — a fictional kitchen-appliance brand selling into the UK and Germany; figures are illustrative, not measured. Version one scored 40 answers at a mean of 8.4. A human review of 12 found five calling the UK plug “European standard”, three quoting the retired warranty, and two stating 1,500 W as 2,000 W. The model scored all ten above 8: structured, cited, confident.
The rebuild made accuracy a gate sourced only to master data and the policy page, deleted two non-discriminating dimensions, and attached the warranty page to the grading context. The mean fell to 6.1; model-human disagreement fell from 14 items to three. A lower score with a more believable conclusion is the correct first result.
The scoring unit follows the answer unit: with no conclusion sentence, the model assembles one from description, and the low score reflects a missing liftable paragraph — the answer unit framework covers that repair.
Where this breaks
Five situations. Where category facts cannot be checked externally — custom, handmade, experience-led goods — accuracy collapses into self-assessment. Without a frozen question set and sampling discipline, every round scores a different object; build the AI visibility baseline first. Below 20 answers a month, two people reading everything is cheaper. Once a score attaches to a person’s performance, it becomes a writing exercise. In non-English markets, run the human pass before trusting verdicts, or a translation weakness gets recorded as a content failure. This is measurement and growth work: the output is a “fix this next” call, not a score.
This month, pick one action: pull 30 verbatim answers from the latest round, have two people score them independently against the five criteria, and list every disagreeing item. If more than a third disagree, rewrite the band definitions before touching the prompt.