Running Three AI Models Instead of One Won’t Save You as Often as You Think

Why multi-model AI reliability doesn’t scale the way pairwise comparisons suggest

Enterprises building routers, cascades or voting systems across multiple models typically select their model pool using pairwise error correlation — checking whether two models tend to fail on the same prompts. If a coding-strong model and a SQL-strong model rarely fail on the same query, the assumption is that combining them behind a router produces a system that rarely fails at either task. The study’s authors, whose findings were reported to VentureBeat, found this reasoning breaks down at exactly the point enterprises rely on it most: predicting how often every model in the pool will be wrong at once.

Tested on the open-ended MATH-500 benchmark, standard pairwise correlation statistics predicted the full 67-model pool would fail together on 2.3 per cent of questions. The measured co-failure rate was 5.2 per cent — roughly 2.25 times higher than the correlation-based estimate. The researchers attribute the gap to what they term a common-mode atom: a slice of queries that the entire model market struggles with simultaneously, which pairwise statistics cannot detect because they only measure how pairs of models relate to each other, not how the whole pool behaves on the hardest cases. Adding more models to the pool does not close this gap, because the shared blind spot is not diversity-related — it is capability-related.

Format matters too. When graduate-level science questions from the GPQA benchmark were converted from multiple-choice to free-response, the share of questions where every model in the pool failed together jumped from near zero to 12.7 per cent. Open-ended generation tasks — the category where enterprises most want a diversity dividend from multi-model orchestration — are exactly where the co-failure ceiling bites hardest.

The hidden cost side of multi-model AI orchestration

Model routers, cascades and Mixture-of-Agents architectures all carry what the researchers describe as a shadow price: added system latency, ongoing infrastructure and governance overhead across multiple API providers, and integration complexity that has to be maintained as each underlying model is updated. Enterprises take on that overhead on the expectation that a diversity dividend — a meaningful accuracy gain from combining models — will show up later and justify the cost. The study’s central finding is that it often doesn’t, because today’s leading models increasingly agree with each other, and, more importantly, tend to fail on the same difficult queries rather than different ones.

There is a further complication for buyers assuming that more models automatically means more safety. When models in a voting pool are not closely matched in capability, naive majority voting can actively underperform a single strong model, because weaker models can outvote a stronger one on the questions that matter most. The researchers’ guidance is to combine only models within a matched capability band — and where that is not possible, to spend the budget on the single best available model rather than building a router around mismatched ones.

A free reliability check before an AI agent vendor decision gets made

The study also offers enterprises a practical, low-cost way to test a vendor’s multi-model reliability claims before committing to an architecture: a statistical technique called a Clopper-Pearson bound. Applied to a held-out set of representative test cases — a set of prior support tickets, inspection reports, or operational queries specific to the buyer’s own environment — it calculates a mathematically guaranteed worst-case ceiling on how often the full model pool could fail together, correcting for the fact that a small sample will always understate the true failure rate. A pool that appears to fail together on only two out of fifty test questions could, once the bound is applied, have a true co-failure rate as high as 12 per cent — a materially different number for anyone using that pool for safety-relevant or compliance-relevant decisions.

Crucially, this test costs nothing beyond queries the enterprise would likely run anyway during vendor evaluation, and it applies most cleanly wherever there is a checkable right answer — a control-room threshold breach, a compliance-format report, a structured maintenance ticket. Where outputs are open-ended and cannot be checked against a defined answer, the study’s authors note that whether their findings generalise remains an open question, since their benchmarks focused on definitively checkable tasks rather than subjective ones.

What the co-failure ceiling means for AI agent vendor evaluation on private networks

For industrial buyers, this research lands directly on a claim increasingly made in AI agent vendor pitches: that a multi-model or ensemble approach is inherently safer than relying on a single model, because the models supposedly cover each other’s blind spots. The evidence suggests that claim needs to be substantiated, not assumed — particularly for the kind of definitively checkable, high-consequence tasks common across manufacturing, mining, ports, airports and utilities, such as extracting a compliance figure from an inspection report, generating a control command that must execute without error, or formatting a structured maintenance or safety log.

Two practical implications follow. First, for tasks with a clear right answer, a well-validated single frontier model may be a more defensible and lower-overhead choice than a multi-model architecture — the added latency, cost and multi-vendor governance risk of orchestration is easier to justify when there is genuine evidence of a diversity benefit, not just a lower pairwise correlation score. Second, where a vendor does propose a multi-model or ensemble architecture as part of an AI agent solution, buyers evaluating that proposal against a private network deployment can reasonably ask for the vendor’s own co-failure testing on a representative sample of the buyer’s operational data — not a generic benchmark score, and not a pairwise correlation figure alone. The measurement is inexpensive to produce; a vendor unable or unwilling to produce it is, in effect, asking the buyer to take the reliability claim on faith.