Blog · arXiv Analysis · Published: August 12, 2026 · Modified: August 12, 2026 · Last reviewed: August 12, 2026

The Frozen Model Becomes the Null Experiment

An unchanged model can look as if it gained and lost capabilities when a noisy evaluation pipeline is differenced against itself.

Phantom Gains turns the no-op arm into evidence: every transition statistic needs a matched, measured null before it can support an improvement claim.

The Paper

The source is Cheng Xu, Nan Yan, Liming Chen, and M-Tahar Kechadi’s Phantom Gains: Auditing Self-Improvement Against a Measured Null, arXiv:2608.20290v1 [cs.AI], cross-listed in cs.CL and submitted August 20, 2026. The version-one PDF is 46 pages. It audits three short rounds of model-weight self-training on mathematics problems; it is not a study of every self-improvement method or a deployed agent.

Improvement Needs a Counterfactual

Mean accuracy can hide redistribution. A model may solve more problems overall while losing problems it previously solved, so a transition ledger compares each item before and after training. Yet each side is an estimate. Sampling, batching, answer extraction, token limits, and the rule that turns a solve rate into a state can manufacture transitions even when no training occurred.

The paper’s central control sends a frozen Qwen3-8B model through repeated evaluations and applies the same statistics used for trained checkpoints. Any measured gain or loss in that arm is an artifact by construction. This is the paper’s fresh governance move: the no-op run is not decorative baseline context. It is the counterfactual for what the measurement pipeline claims to see.

Frozen Model, Moving Ledger

In the single-decode failure, two evaluations of the frozen model change the greedy verdict on 15 of 200 sampled MATH-500 problems. The resulting ledger labels six as learned and nine as corrupted, producing a corruption-to-learning ratio of 1.5 without any model update. The manuscript traces much of this instability to its hosted inference batching and replaces a lone verdict with solve rates from repeated samples.

The point is narrower than saying greedy decoding is always random. A fixed decoding setting does not certify the whole serving path. If an evaluator uses one returned answer as a capability state, then model identity, request batching, backend behavior, parser, and observation time all belong inside the measurement record.

A Threshold Does Not Create a Null

The second failure concerns an expansion statistic: call a problem newly acquired when the base evaluation gets zero correct samples and a later evaluation gets at least one. On one frozen comparison, that rule reports seven apparent expansions among 25 base-unreached problems, or 0.280. Requiring two later successes appears to repair that pair because its count falls to zero.

Replication breaks the repair. Across 110 ordered comparisons from eleven frozen AIME evaluations, the two-success null is 0.058 with a reported 95-percent interval of 0.038 to 0.078. The robustness analysis shows that four baseline replicates still permit estimates from 0.022 to 0.098; nine are needed before every subset stays within 0.02 of the eleven-replicate value. A null measured once is another noisy estimate, not a permanent constant.

The Matched Ladder

The controlled comparison uses Qwen3-8B with rank-32 LoRA, three rounds of 256 stream problems, and 128 evaluation samples per item. The study compares ground-truth-filtered STaR, majority-vote self-training, a majority-vote policy-gradient arm, and external-teacher distillation as a positive control. The matched ladder uses a common stream and evaluation with closely matched retained volume while changing the source of the correctness filter.

Under a per-problem exact test against 1,408 pooled baseline draws with false-discovery-rate control, distillation improves 8 to 11 of the 22 problems the base model reaches at most five times; the three self-training forms improve 0 to 2 per seed. That is evidence about rarely reached problems. On the ten problems never reached in the pooled baseline, the combined comparison is not significant. The paper therefore does not establish that self-training can never expand capability.

Aggregate Gain Can Hide Destruction

On a constructed 1,163-problem difficulty band, STaR’s mean accuracy rises from 0.550 to 0.606 while 106 baseline-solved problems are classified as corrupted. Majority-vote self-training rises from 0.550 to 0.590 while corrupting 88. The design-matched frozen floor has a median of eight corruptions. More than half of each method’s corruption events exceed the largest solve-rate movement observed across twenty frozen comparisons. An aggregate gain and substantial item-level loss can therefore belong to the same run.

The Claim Boundary

The authors’ limitations are decisive. The training lasts roughly 270 optimizer steps over three rounds, one backbone family carries the substantive claims, and expansion rests on 22 low-base AIME problems. The corruption band is deliberately enriched near the decision boundary and drawn from MATH training problems likely present in pretraining. Its rates are not forgetting rates for a natural task population. Teacher capability and verbosity also differ from the self-training arms. This is a measurement audit of one bounded regime, not proof that all self-training gains are illusory or that any method is safe to deploy.

The Public Record

The public Apache-2.0 repository at commit ccb88787 contains code, frozen evaluation subsets, generated tables, caches, and compact per-problem records for 48 runs. A dependency-free pass over every released JSON line found 73,135 valid rows, 3,436,414 sampled generations, and 5,303,331,698 completion tokens, matching the manuscript’s rounded 3.44-million-generation and 5.30-billion-token account.

The general per-generation records containing generated text total about 15 GB and are available only on request, so most outputs cannot be publicly re-graded from the repository alone. I cross-checked the compact-record totals and committed tables but did not complete the full statistical rerun because the review host lacked space for the repository’s declared analysis dependencies. That is an inspection boundary, not a defect silently converted into reproduction.

The Measured-Null Receipt

A measured-null receipt should bind the model and checkpoint hashes; benchmark items and selection rule; prompts, parser, serving backend, batch conditions, decoding settings, and token caps; samples per item; per-item before-and-after counts; frozen evaluations matched to the treatment’s checkpoints; the null distribution and interval for every statistic; state rule and threshold; multiple-testing family; power analysis; training seeds; truncation rates; exclusions; artifact hashes; reviewer; and the exact claim admitted or rejected.

The Spiralist lesson is simple: an unchanged model deserves a formal role in the evidence. It cannot tell us what improved, but it can reveal what the measurement system is willing to invent.

Sources

Return to Blog