Blog · arXiv Analysis · Last reviewed July 10, 2026

The Evaluation Budget Becomes the Stopping Rule

Ofir Arviv, Kristjan Greenewald, Yotam Perlitz, Hadar Mulian, Michal Shmueli-Scheuer, and Leshem Choshen's July 2026 arXiv paper asks when model evaluators should stop spending examples and start making a statistically stated decision.

Published: July 10, 2026 Modified: July 10, 2026 Last reviewed: July 10, 2026 Tags: arXiv, AI governance, evaluation, benchmarks, sequential testing, model selection

The Paper

The paper is Stop Guessing When to Stop Testing: Efficient Model Evaluation with Just Enough Data, arXiv:2607.08522 [cs.LG]. The arXiv record lists Ofir Arviv, Kristjan Greenewald, Yotam Perlitz, Hadar Mulian, Michal Shmueli-Scheuer, and Leshem Choshen as authors, with version 1 submitted on July 9, 2026. The PDF is 11 pages, the arXiv HTML lists IBM Research, and the authors point to the public OfirArviv/adaptive-eval repository.

The paper belongs beside benchmark-as-agent work, judge drift measurement, live benchmark maintenance, active learning, and AI audit trails. Its fresh angle is not a new score. It is the stopping rule behind the score.

The Fixed-Benchmark Problem

Most model evaluations begin by choosing a fixed sample size. That habit is easy to operationalize and hard to defend. A development team may only need a rough signal while debugging. A deployment team may need evidence that a candidate is practically and statistically better than the incumbent. A leaderboard may need to separate close models without spending the full benchmark on every unpromising pair.

The paper frames fixed-size evaluation as inefficient in both directions. It can spend examples after a result is already clear, or it can stop with too little evidence while still producing a number that looks decisive. The cost pressure is rising because modern evaluations can involve high-resolution images, long contexts, many model comparisons, expensive task instances, and LLM-as-judge calls. The score is cheaper to read than to earn.

Sequential Testing

Arviv and coauthors treat evaluation as a sequential statistical decision. The evaluator declares an objective, runs an initial batch, checks whether the stopping criterion has been met, and either stops or samples another batch. The paper studies criteria such as confidence-interval width, diminishing returns, and minimum detectable or practically meaningful effect size.

This is not just "look early and quit when happy." The paper explicitly discusses the peeking problem: repeatedly inspecting interim results can invalidate ordinary p-values and confidence intervals. Its pairwise-testing setup uses group sequential testing, the gsDesign R package integrated through Python, and a Pocock spending function so repeated looks are part of the design rather than an after-the-fact convenience.

What the Paper Found

The main empirical setting uses Open VLM Leaderboard data from VLMEval/OpenVLMRecords commit dbc5e10, covering 206 vision-language models across 31 multimodal benchmarks. The paper reports that an adaptive confidence-interval stopping rule can reduce evaluation cost by up to 80 percent when a 2.5-point interval-width allowance is acceptable. A diminishing-returns rule saves 44 percent of the cost while sacrificing 0.132 points of precision in the reported setup.

One concrete example makes the receipt legible: at 8,000 examples, about 55 percent of the full 14,400-example dataset, the confidence interval width is 2.9 points versus 2.7 points on the full dataset. That difference may matter for very close model rankings. It may be immaterial for a coarse development decision. The paper's point is that the acceptable loss should be stated before the score is used.

For pairwise comparison, the authors analyze 1,000 model pairs drawn from the top 50 models. A fixed 1,200-sample comparison reliably differentiates 55 percent of pairs at 95 percent confidence; the adaptive framework differentiates 76 percent under the reported settings. Model pairs separated by more than 2 points usually save at least 60 percent of the evaluation effort. The case studies report 60 percent, 63 percent, and 30 percent sample use in different evaluation goals, with the model-selection case filtering out 86 weak models.

Limits

The paper is careful about scope. Its procedures assume independent and identically distributed samples more often than messy benchmark curation may justify. Sequential methods add overhead, and repeated runs can vary unless the randomization and batch schedule are controlled. The authors also note that choosing the right stopping criterion requires both statistical and domain expertise.

The public repository documents the paper-facing implementation, data layout, scripts, and configuration paths. That is useful, but it does not turn the result into a generic governance machine. An organization still has to say what it is optimizing: cost, precision, deployment confidence, regression detection, or candidate screening.

The Spiralist reading is that a benchmark is not a sacrament. It is a budgeted measurement ritual. If the stopping rule is hidden, the final score arrives without its conditions of production.

The Receipt

An evaluation-budget receipt should record the benchmark version, dataset source, task mix, scoring function, model pool, comparison objective, initial sample size, batch size, random seed, confidence level, interval-width threshold, minimum effect size, alpha-spending rule, stopping reason, number of examples actually used, full-run fallback rule, indistinguishable comparisons, compute or annotation cost, and the decision that the score was allowed to support.

The audit question is not "what was the benchmark score?" It is "what question was the evaluation allowed to answer, and why was it statistically legitimate to stop there?"

Sources

Return to Blog