The Same Accuracy Score Can Hide Different AI Decisions

In short: two AI configurations can post the same accuracy score and still give different answers on many individual cases. When the cases they disagree on matter to a decision, the overall score tells you little about which configuration you can rely on. An assurance record needs the configuration and model version, how uncertain the reference labels are, the error slices that matter to the decision, and the events that trigger a retest.

This guide explains a November 4 research preprint on the problem, works through a synthetic example, and ends with a candidate assignment, a recruiter question and a one-page retesting sheet you can download. It extends Projects 1 and 2 of our AI Assurance Engineering Skills Map.

What the Research Found

Louise Yorke of University College London compared eight GPT-4o configurations that classified 1,200 UK Prime Minister's Questions into 21 policy topics, using human-coded labels from the Comparative Agendas Project as the reference [1]. The configurations varied things a team might change without much thought: the output format, few-shot examples, whether the model reasons before answering, and how the prompt scopes the task.

Accuracy against the human coding ranged from 68.2% to 70.3%, and none of the differences was statistically significant after correction for multiple comparisons. Yet 291 of the 1,199 questions with complete labels (24.3%) received different labels across the configurations. For 215 questions, the label matched the human coding under some configurations and not others (Section 4.1.2 of [1]).

Two more findings matter for anyone who signs off on AI evidence. First, the configurations agreed with each other more than with the human coders. For 191 questions, all eight gave the same label and that label differed from the reference. Because all eight used one model family, their agreement is not independent evidence that they were right (Section 4.1.3). Second, the default configuration did not give identical answers when rerun with unchanged settings: pairs of runs disagreed on 2.78% of labels on average, even at temperature zero (Section 4.1.1).

What the study does not show. It covers one task, one model family and a small set of configurations. The reference labels are human judgments and can be wrong. The repeat runs came about five months after the original experiment and used an unpinned model alias, so the author says they cannot separate configuration effects from change over time (Section 6). The 24.3% figure is a disagreement rate across configurations on this task. It is not an error rate, and it is not a benchmark for AI systems in general.

A Synthetic Worked Example

Everything in this section is synthetic. Every number below was invented for teaching. They do not come from the study or from any real deployment.

The fictional Lakemont Benefits Office uses a language model to sort incoming letters into three queues: urgent hardship, complaint and routine request. Urgent hardship letters go to a caseworker within one business day. The team tested two configurations against 200 letters that staff had labeled by hand.

• Configuration A: model snapshot M-2026-09 (pinned), prompt P-3, temperature 0, label-only output.
• Configuration B: the same snapshot, prompt P-4 with five worked examples added, temperature 0, label-only output. B is faster and cheaper because P-4 lets the team drop a second review pass.

Both configurations score 70%. They give different labels on 38 of the 200 letters (19%). On 16 of those, only A is right; on another 16, only B is right; on the remaining 6, both are wrong in different ways. On 162 letters they agree, and 38 of those agreed labels are still wrong. Staff coders had disagreed with each other on 18 letters before a supervisor settled the label, and 7 of those 18 are among the letters where A and B differ.

The decision that changes. The office's own release rule (also synthetic) says a configuration may not go live unless it correctly labels at least 75% of urgent hardship letters in the test set. On the overall score alone, the team would pick B because it is cheaper. On the slice that the rule protects, A passes at 80% and B fails at 50%. The conclusion about which configuration is fit to release flips, and the overall accuracy never moved.

Two cautions keep the example honest. Thirty urgent letters is a small sample, so the team should report the count with the percentage and treat a near-miss as a reason for more testing, not a pass. And because 7 of the disputed letters had uncertain reference labels, someone should review those letters before anyone claims either configuration is right on them.

What an Assurance Record Should Capture

• Exact versions. The dated model snapshot, never only a moving alias such as latest; the prompt version; decoding settings; output format and parsing rules; any retrieval index or tool configuration; the evaluation dataset version; and the date of the run.
• Reference-label uncertainty. Who labeled the reference set, how disagreements between labelers were settled, how many labels were disputed and where those cases fall in the results.
• Material error slices. The groups of cases where an error has a different consequence, defined before the test is run and tied to a decision rule. Report counts as well as percentages.
• Disagreement between configurations. When comparing configurations, record how many cases changed label and which slices they fall in. Note how much the same configuration varies when rerun unchanged, so you can tell configuration effects from noise.
• Review and retest triggers. The events that make this evidence stale for the decision it supports. The sheet below lists a starting set.

When to Retest

Retest the affected slices, and the overall score with them, when any of these change: the model snapshot or provider version (including a silent change behind an alias); the prompt, examples or output format; decoding settings; the parsing or score-extraction code; the reference labels or labeling guidance; the mix of incoming cases; the decision rule or threshold; or the people and process that act on the output. Also retest when monitoring shows a shift in label shares, a rise in overrides by staff, or complaints that point to one slice. No source sets universal thresholds for these triggers; the owner has to choose them and write down why.

Who Does This Work

Technical assurance and evaluation staff design the test and produce the slice results. Governance and QMS staff make sure the record names versions, owners and triggers, and that a changed configuration cannot reuse old evidence without review. Model risk and validation staff challenge whether the reference set and slices are adequate. Internal audit checks, after the fact, whether the release followed the rule. The AI evaluation and testing skills guide and the model risk management skills guide cover each side in more depth.

Candidate Assignment

Build a synthetic comparison of two configurations of one classifier that land within one percentage point of each other on overall accuracy. Define two material slices before you look at the results and set a decision rule for each. Show the disagreement count, one slice where the configurations differ enough to change the release decision, and the effect of disputed reference labels. Write a one-page decision record naming the versions tested, the configuration you would release and why, what remains uncertain and three retest triggers. Label every number as synthetic. A spreadsheet is enough; code is optional.

Recruiter Question

"Two configurations of our document classifier both score 70% accuracy. The engineering team wants to ship the cheaper one. What do you need to see before you agree?"

Indicators of a strong answer:

• Asks how many cases the two configurations label differently, and where those cases fall.
• Asks which errors cost the most and whether a slice-level rule was set before testing.
• Asks for exact model and prompt versions and whether either result came from an unpinned alias.
• Questions the reference labels: who made them, how disputes were settled, how many are uncertain.
• Does not treat agreement between the two configurations as proof that either is right.
• Names a decision owner and the conditions that would bring the decision back for review.

A weak answer picks the cheaper option because the scores match, or asks only for a bigger test set without saying what it would look for.

Download the Retesting Sheet

The one-page retesting reference sheet lists the record fields, the slice checks and the retest triggers from this guide in a form you can print or attach to a release record. It is an editorial checklist, not a standard.

Download the retesting sheet (PDF, one page)

Sources and Free Learning

[1] Louise Yorke, "Beyond accuracy: what AI assurance needs to establish in public services," arXiv preprint 2611.06655, version 1, submitted November 4, 2026. https://arxiv.org/abs/2611.06655
A research preprint with free full text. Read Table 1 and Sections 4.1 and 6. The paper also reviews 37 UK and EU policy documents on testing and oversight; that part is the author's analysis, not official guidance.

[2] National Institute of Standards and Technology, AI Risk Management Framework page. https://www.nist.gov/itl/ai-risk-management-framework
Free. The current version is AI RMF 1.0 (January 26, 2023), which is voluntary. NIST's page says it is being revised; no new edition had been released when we checked on November 7, 2026.

The worked example, the decision rule and the assignment are GRC Careers editorial material. The Lakemont Benefits Office and all of its numbers are synthetic.