Why do small LLM benchmarks report inaccurate differences?

Accuracy on a benchmark is a proportion measured on a finite sample, meaning it carries sampling noise. Small benchmarks lack the statistical power to resolve small differences; any gap below the minimum detectable effect is mathematically indistinguishable from random noise.

Why should I use McNemar's test for benchmark comparisons?

When comparing two models on the exact same benchmark items, the evaluation is paired. McNemar's test isolates the discordant items (where the models disagree), effectively ignoring the items both got right or wrong. This dramatically increases statistical power, requiring far fewer items to prove a difference.

Power Analysis for Benchmark Design

Benchmarks report differences they cannot detect

A leaderboard reports that one model scores 87.2% and another 86.1%, and a great deal is read into that 1.1-point gap. But each score is an estimate from a finite set of items, and if you reran the same models on a fresh sample of the same size, both numbers would move. The question that decides whether the gap means anything is whether the benchmark is large enough to tell a real difference from that sampling noise, and it usually is not. Card et. al made this case for the field in "With Little Power Comes Great Responsibility" (2020): statistical power has been largely ignored in NLP leaderboards, underpowered comparisons are common, and, to take their example, a typical machine-translation test set of 2000 sentences has only about 75% power to detect a difference of one BLEU point (this is a common NLP scoring metric). Miller made the same argument for modern language-model evaluations in "Adding Error Bars to Evals" (2024). The tools to check this are standard and well known to those setting up A/B tests, and these should be used for benchmark evaluations too.

Accuracy is a proportion

Accuracy is usually defined as the fraction of items a model gets right, making it a proportion estimated from $n$ trials. Comparing two models is therefore comparing two proportions. Three core statistical quantities are important for this task:

Significance level ( $\alpha$ ): the false-positive rate tolerated when detecting a difference, conventionally set to $0.05$ .
Power ( $1-\beta$ ): the probability that a true difference of a given size actually trips the statistical test, conventionally set to $0.80$ .
Effect size: the absolute difference in accuracy we care to resolve, for example a $2$ -point gap.

Power analysis ties these three together alongside the sample size $n$ : fix any three and the fourth is determined. For two proportions the relationship has a closed form resting on a normal approximation, shown below, which we also check against simulation.

Items needed to detect

p_1

p_2

(two-sided)

n = \dfrac{\left(\, z_{\alpha/2}\sqrt{2\bar p\,(1-\bar p)} \;+\; z_{\beta}\sqrt{p_1(1-p_1)+p_2(1-p_2)} \,\right)^{2}}{\left(p_2-p_1\right)^{2}}

where

\bar p=(p_1+p_2)/2

, and

z_{\alpha/2}=1.96

\alpha=0.05

z_{\beta}=0.84

at 80% power.

import numpy as np
from scipy.stats import norm

def items_needed(p1, p2, power=0.80, alpha=0.05):
    za, zb = norm.ppf(1 - alpha/2), norm.ppf(power)
    pbar = (p1 + p2) / 2
    num = za*np.sqrt(2*pbar*(1-pbar)) + zb*np.sqrt(p1*(1-p1) + p2*(1-p2))
    return (num / (p2 - p1))**2                  # items per model (same items => benchmark size)

def power_at(p1, p2, n, alpha=0.05):
    za = norm.ppf(1 - alpha/2); pbar = (p1 + p2) / 2
    se0 = np.sqrt(2*pbar*(1-pbar)/n); se1 = np.sqrt(p1*(1-p1)/n + p2*(1-p2)/n)
    return norm.cdf((abs(p2-p1) - za*se0) / se1)

# validate the closed form by simulating the actual two-proportion test
rng = np.random.default_rng(0)
def simulated_power(p1, p2, n, alpha=0.05, trials=40000):
    x1, x2 = rng.binomial(n, p1, trials), rng.binomial(n, p2, trials)
    ph1, ph2 = x1/n, x2/n; pbar = (x1 + x2)/(2*n)
    se = np.sqrt(pbar*(1-pbar)*(2/n))
    z = (ph2 - ph1) / np.where(se > 0, se, 1e-9)
    return np.mean(np.abs(z) > norm.ppf(1 - alpha/2))

n = int(np.ceil(items_needed(0.85, 0.87)))
print(f"detect 85% vs 87% at 80% power: {n:,} items")
print(f"closed-form power at that n: {power_at(0.85, 0.87, n):.3f}")
print(f"simulated  power at that n: {simulated_power(0.85, 0.87, n):.3f}")

detect 85% vs 87% at 80% power: 4,724 items
closed-form power at that n: 0.800
simulated  power at that n: 0.802

How many items do you need?

Running the formula across a range of baseline accuracies and target differences produces a reference table. Each cell is the number of items needed to detect that difference at 80% power; for a shared benchmark, that count is simply the benchmark's size.

for p in [0.50, 0.70, 0.85, 0.95]:
    row = "  ".join(f"{int(np.ceil(items_needed(p, p+d))):>6,}" for d in [0.01, 0.02, 0.03, 0.05])
    print(f"baseline {p:.2f}: {row}")

Items needed to detect a difference at 80% power (two-sided, α = 0.05)

baseline accuracy	+1 point	+2 points	+3 points	+5 points
50%	39,240	9,806	4,356	1,565
70%	32,647	8,080	3,554	1,251
85%	19,461	4,724	2,036	686
95%	6,745	1,506	588	152

The numbers are larger than most people expect. Detecting a two-point difference at 80% power takes on the order of ten thousand items near 50% accuracy, and still a few thousand even up near 95%, where the variance of a proportion is smallest. When it comes to Language Models, very few real benchmarks are that large, which is the whole problem, and it is easier to see the other way around.

What can your benchmark actually resolve?

Instead of asking how many items you would need, fix the benchmark's size and solve for the smallest difference it can detect at 80% power. This minimum detectable effect is the resolution floor of the benchmark: any reported gap below it is within the noise band, and a single run cannot tell it apart from zero.

from scipy.optimize import brentq

def min_detectable(n, p1=0.50, power=0.80, alpha=0.05):
    hi = min(0.5, 0.999 - p1)
    if power_at(p1, p1 + hi, n) < power:
        return float('nan')                       # no gap of any size reaches 80% power
    return brentq(lambda d: power_at(p1, p1 + d, n) - power, 1e-6, hi) * 100

for name, n in [("AIME 2024", 30), ("HumanEval", 164), ("GPQA Diamond", 198),
                ("SWE-bench Verified", 500), ("GSM8K", 1319), ("MATH", 5000), ("MMLU", 14042)]:
    print(f"{name:20} {n:6,} items  ->  cannot resolve below +/- {min_detectable(n):.1f} pt")

AIME 2024                30 items  ->  cannot resolve below +/- 33.4 pt
HumanEval               164 items  ->  cannot resolve below +/- 15.2 pt
GPQA Diamond            198 items  ->  cannot resolve below +/- 13.9 pt
SWE-bench Verified      500 items  ->  cannot resolve below +/- 8.8 pt
GSM8K                 1,319 items  ->  cannot resolve below +/- 5.4 pt
MATH                  5,000 items  ->  cannot resolve below +/- 2.8 pt
MMLU                 14,042 items  ->  cannot resolve below +/- 1.7 pt

Baseline 50% is the worst case, since a proportion's variance peaks there; at accuracy $a$ the floor scales by $2\sqrt{a(1-a)}$ , so the gaps only shrink from here.

Resolution improves only with the square root of the number of items, so buying finer resolution gets expensive fast: cutting the detectable gap in half costs four times the items. The smallest benchmarks sit at the top of the chart with the longest bars, where a 30-item set like a single year of AIME cannot separate models less than about thirty points apart, and a 164-problem set like HumanEval cannot see below roughly fifteen. Only once a benchmark reaches the thousands does its floor drop under a handful of points, and reported sub-point differences on anything smaller are noise being read as signal.

Paired designs: only disagreements count

There is good news, and it comes from a design choice most benchmark comparisons already make without exploiting it. When both models are run on the same items, the comparison is paired, and the two-independent-proportions test above throws away the pairing and so overstates how many items you need. The right test for shared items is McNemar's: items where both models are correct, or both wrong, tell you nothing about which is better, so only the discordant items, the ones where the models disagree, carry any information. The effective sample size is not the benchmark's size but the number of disagreements, and since strong models agree on most items, that can be a small fraction of the whole.

Paired items needed, in terms of the disagreement rate

d

n = \dfrac{\left(\, z_{\alpha/2}\sqrt{d} \;+\; z_{\beta}\sqrt{d-\delta^{2}} \,\right)^{2}}{\delta^{2}}

where

d

is the fraction of items the two models score differently and

\delta

is the true accuracy gap. Only the disagreements enter, so the required count tracks

d

, not the benchmark's size.

def items_needed_paired(disagree_rate, delta, power=0.80, alpha=0.05):
    # disagree_rate = fraction of items the two models score differently; delta = true accuracy gap
    za, zb = norm.ppf(1 - alpha/2), norm.ppf(power)
    return (za*np.sqrt(disagree_rate) + zb*np.sqrt(disagree_rate - delta**2))**2 / delta**2

print(f"detect a 2-pt gap, unpaired: {int(np.ceil(items_needed(0.80, 0.82))):,} items")
for agree in [0.75, 0.85, 0.93]:
    n = items_needed_paired(1 - agree, 0.02)
    print(f"  models agree on {agree:.0%} of items -> paired needs {int(np.ceil(n)):,} items")

detect a 2-pt gap, unpaired: 6,039 items
  models agree on 75% of items -> paired needs 4,904 items
  models agree on 85% of items -> paired needs 2,941 items
  models agree on 93% of items -> paired needs 1,372 items

The more the two models agree, the fewer items the paired test needs, because the signal lives entirely in the disagreements. Two models that agree on 93% of a benchmark are compared, in effect, on the remaining 7%, and the paired analysis needs roughly a fifth of the items the unpaired one demands. The practical lesson is to always evaluate both models on the same items and analyze the result with McNemar's test rather than comparing two independent scores.

What to do about it

None of this requires new tooling. Before trusting a difference, check the minimum detectable effect; if the gap is smaller, the correct conclusion is that the benchmark cannot tell the models apart, not that they are equal. Report an interval, not a bare number: a Wilson or Clopper-Pearson confidence interval on each accuracy makes the uncertainty visible at a glance. When comparing two models, run them on the same items and use McNemar's test to claim the paired power. And when you are designing a benchmark, work backward from the smallest difference you need to resolve to the number of items it demands. The uncomfortable corollary is that a lot of published one and two-point improvements on small benchmarks are not measurable with the benchmark that produced them.

Common questions

My benchmark has fewer items than the table demands. Is it useless?

Not useless, just limited to coarser differences. A small benchmark can still resolve large gaps with confidence; it simply cannot resolve small ones. The chart tells you where its floor is, and the right response is to report differences relative to that floor and to avoid claiming precision the sample size does not support. Aggregating several small benchmarks, or evaluating on the same items with a paired test, are the usual ways to recover power.

Why is the baseline accuracy in the formula, and which one should I use?

Because a proportion's variance is $p(1-p)$ , which is largest at 50% and shrinks toward the extremes, so the same difference is easier to detect when accuracy is near 0% or 100% than near the middle. The chart uses 50% as the worst case; for a benchmark where models sit around accuracy $a$ , the resolution floor scales down by $2\sqrt{a(1-a)}$ , so a benchmark saturated near 95% resolves somewhat finer differences than the worst-case line shows.

Does higher power just mean I need more items?

Yes, and the cost is steep because it enters through the square root. Moving from 80% to 90% power raises the z multiplier from 0.84 to 1.28, and moving to 95% raises it to 1.64, each of which inflates the required item count. Eighty percent is the common convention because it balances the two error rates reasonably, but the point of running the calculation is to make that trade explicit rather than to discover after the fact that an experiment never had a chance of seeing the effect.

References

Card et. al, "With Little Power Comes Great Responsibility" (2020) — power analysis across NLP; underpowered comparisons are common.
Miller, "Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations" (2024) — sample-size and error-bar methods for LM evaluations.
McNemar, "Note on the sampling error of the difference between correlated proportions or percentages" (1947) — the paired test for shared items.