Computing & AIPreprintExperiment5 min read

THIRTY AIs, ONE MISTAKE

The wisdom of crowds, an idea the paper traces back to Galton in 1907, says: average many independent guesses, and the errors cancel. If the errors are unbiased (on average they lean neither too high nor too low) and independent, the error of the average shrinks like one over the square root of the number of guesses. Many systems that combine several AI language models — by sampling and voting, by debate, or by committees of different models — lean on this argument: a bigger team can stand in for a better agent.

Carolina Fortuna and Blaž Bertalanič, at the Jožef Stefan Institute in Ljubljana, point out that the argument is only as good as its premises, and set out to test them.

Fermi questions as a test bench

Fermi questions ask for quantities that can only be estimated, such as how many objects fit in a container. Their answers are numbers spanning many orders of magnitude, so an error can be measured in orders of magnitude — a factor of ten counts as one — and errors of opposite sign can genuinely cancel.

The team collected 2.51 million answers from 15 open-weight models (models whose trained parameters are published), from 3 to 72 billion parameters, to the 529 questions of the realFP benchmark, and checked the results on two other sets of questions. Each model answered each question about 350 times, and teams of 2 to 30 answers were averaged on a logarithmic scale, that is, by averaging their orders of magnitude rather than the raw numbers.

On their own, the models are poor at this game. A single answer has the right order of magnitude on 12 to 34% of the questions, and is off by more than a factor of a hundred on 30 to 63%.

The error every copy shares

The key step is to split each answer’s error into two parts. One is the mistake a model makes on a particular question, shared by all its answers to it. The other is the random part that varies from one answer to the next. Only the second can cancel when answers are averaged.

It turns out the first part dominates. On realFP, it accounts for a median 0.906 — about nine tenths — of the error variance, a measure of how widely the errors scatter. On a given question, the random parts of two answers from the same team are unrelated, as independent sampling implies; what ties them together is the mistake they share.

The paper turns this into a single number, with a classic formula known as Kish’s design effect: the effective number of independent answers in a team of N is N / (1 + (N − 1)ρ), where ρ is that shared share. With ρ = 0.906, a team of thirty is worth 1.10 independent agents.

Two panels: on the left, team error barely changes with team size for 15 models; on the right, the same relative to one agent, flat near 1 while theoretical references fall steeply.

Left: team error versus team size N for 15 models. Right: the same relative to a single agent (coloured lines, near 1), against what independent and unbiased answers would give (black dashed and dash-dot lines) and the 1/√N rate (dotted). — Figure 2, Fortuna & Bertalanič (2026), arXiv:2611.16315.

In practice, teams of thirty reduced a single agent’s error by a median 1.7%, and at most 5.4%. Independent answers that also carried no shared mistake on each question would have cut it by 74 to 79%.

Everything drifts towards a million

Part of the shared mistake has a recognisable shape. Every model overestimates small quantities and underestimates large ones, compressing its estimates towards about a million (10⁶). This trend appears in all 15 models, but it explains only a median 17% of the shared error. The models also err on the same questions in the same direction: the per-question errors of two different models are still correlated at 0.60 to 0.69 once extreme cases are tamed and the drift towards a million is removed.

Scatter of errors against the true answer on a logarithmic scale, with a red median line falling from positive errors for small answers to negative errors for large ones.

Median error of the 15 models on each question, against the true answer (powers of ten): small quantities are overestimated, large ones underestimated. — Figure 1 (left), Fortuna & Bertalanič (2026), arXiv:2611.16315.

Talking it over does not help

What if the agents discuss? In three rounds of debate, each agent saw its teammates’ answers and revised. The spread within teams of five collapsed by 93% — by the third round, for 12 of the 15 models, three of five agents gave the same number on at least half the questions. But the shared mistake did not move, and the total gain was 2.1%, at fifteen times the computing cost. Agents that simply revised their own answer, without seeing peers, did at least as well.

What helped a little

Raising the sampling temperature — the setting that controls how much chance enters each answer — made answers more varied without improving teams by more than about 1%. Mixing different models helped more: five models of similar accuracy did 6.6% better together than in their own teams. Yet no mixture of the 15 models improved significantly on a team built from the single most accurate one, gpt-oss-20b. A correction learned on questions with known answers removed the models’ overall bias, but not the question-by-question mistake.

The authors stress the limits: open models of modest size, without their “thinking” modes, on these benchmarks and prompts; frontier and reasoning models were not tested. Their conclusion is a design lesson. Adding more agents, or more disagreement, does not fix a mistake that every copy of a model shares. The authors suggest attacking that shared error directly, for instance by giving agents information specific to each question — a route they have not yet tested.