Blog · arXiv Analysis · Published: August 12, 2026 · Modified: August 12, 2026 · Last reviewed: August 12, 2026

The Option Order Becomes the Benchmark Confound

A multiple-choice score can mix knowledge with sensitivity to where an answer appears.

Removing that positional influence does not necessarily recover knowledge, and improving accuracy does not necessarily reveal which bias was removed.

The Paper

The source is Karl Hanna and Chen Feng's Accuracy and Order Sensitivity Diverge Under Label-Free Strategies, arXiv:2608.11947v1 [cs.CL, cross-listed cs.AI], submitted August 12, 2026. The authors test six language-model configurations on 1,000 sampled MMLU questions and 1,000 sampled ARC-Challenge questions. Their question is unusually useful for governance: if a procedure hides labels during answer commitment or removes joint option position entirely, does it also produce more correct answers?

Two Problems, Not One

Option-order sensitivity and inaccuracy are different defects. A model can give the same wrong answer under every permutation. It can also flip between answers while remaining correct under the one ordering chosen for a leaderboard. The paper measures correctness with accuracy and order sensitivity with per-question flip rate after cyclically rotating options. It also reports recall standard deviation, or RStd, across gold-answer positions, but treats that as an aggregate imbalance diagnostic rather than direct counterfactual evidence about the same questions.

The Label-Free Strategies

The two-stage strategy first shows the model only the question and asks for a free-text answer. A second model call receives that answer plus all options and chooses the closest match. Labels are hidden during commitment, but option order returns during matching, so the procedure is not position-invariant by construction.

The independent-hypothesis strategy instead makes one call per option. Each call sees the question and one candidate answer, assigns a score from zero to 100, and the system selects the highest score, using a seeded random tie-break. No joint option list exists, so display position cannot affect the result. That structural success creates the paper's cleanest test: removing positional influence can be verified even when accuracy does not improve.

What the Results Separate

Across the reported cells, two-stage prompting lowers end-to-end accuracy in 11 of 12 model-benchmark pairs. Independent hypothesis lowers it in eight of 11 valid pairs; Gemini on ARC-Challenge is excluded after provider-side failures left only 313 of 1,000 questions successfully scored. Cyclic permutation, which rotates options, maps answers back to their semantic content, and votes, improves accuracy in 10 of 12 pairs.

One comparison captures the central separation. For GPT-4.1 mini on MMLU, two-stage prompting cuts flip rate from 21.6 percent to 11.8 percent while accuracy falls from 81.8 percent to 80.1 percent. The procedure becomes more stable under option rotation without becoming more accurate. That is not a paradox. It is evidence that robustness and correctness answer different questions.

Comparison Is Part of the Task

The paper's two-by-two diagnostic grid crosses hidden versus visible options in the first stage with LLM versus embedding matching in the second. Hidden options plus embedding matching collapses accuracy; showing options recovers much of the loss. For GPT-4.1 mini on MMLU, hidden-plus-embedding scores 45.8 percent while visible-plus-embedding scores 81.7 percent, close to the 81.8 percent direct baseline. The authors therefore locate the main bottleneck in withholding the choices, not merely in matching prose back to an option.

This matters because multiple-choice options are not always decorative labels attached to independently answerable questions. They can disambiguate what comparison is being requested. Scoring each candidate alone removes a positional variable, but it also removes comparative context. A cleaner interface can therefore measure a different task.

The Diagnostic Can Mislead

RStd looks for unequal recall when the correct answer occupies A, B, C, or D. Yet the positionally invariant independent-hypothesis method still produces nonzero RStd in all 11 valid cells, from 1.1 to 3.9 percentage points. The paper explains why: the position groups contain different questions, so finite samples and difficulty differences can create imbalance without any causal position effect. Under two-stage prompting, Qwen-local's RStd falls on both benchmarks even as its direct flip rate rises. A tidy marginal distribution is not proof that individual answers stopped moving.

Why Cyclic Voting Is Not a Mechanism Test

Cyclic permutation performs best among the model-agnostic methods tested, but it uses one call per option and aggregates several predictions. Its gain cannot be assigned solely to debiasing: voting itself can recover answers. Independent hypothesis costs the same number of calls and loses to cyclic in 10 of 11 valid pairs. The practical result is comparative, not universal. In this experiment, rotating and aggregating is often more accurate than hiding or isolating options; it does not prove that one positional mechanism explains the gain.

The Claim Boundary

The study covers six model configurations, two English benchmarks, mostly four-option questions, one sampled run per condition, and one implementation of each strategy. The flip-rate experiment has no repeated identical-order control, so provider nondeterminism may contribute to flips. The two-stage prompt ablations do not allow Stage 1 reasoning, larger models and other question distributions are untested, and parse failures shrink some scored subsets. The authors publish ChoiceBench code and prompts under an MIT license; I inspected the public repository and paper artifacts but did not rerun paid model calls or independently reproduce the tables.

The Option-Order Receipt

A multiple-choice evaluation should record the dataset revision, sampled question IDs, option count, canonical order, every tested permutation, semantic answer mapping, model and provider version, prompt, decoding settings, repeated-identical-order control, parse failures, scored denominator, accuracy, per-question flip rate, marginal position statistic, confidence interval, vote and tie rules, call and token cost, raw outputs, code commit, reviewer, and correction history. Results should remain split by model and benchmark rather than collapsed into a single claim that a method “debiases” evaluation.

The Governance Standard

The durable rule is to name the property a number actually measures. Position invariance says that rearranging answer choices does not change the semantic prediction. Accuracy says that a prediction matches a key under a defined protocol. Efficiency says what that protocol costs. None entails either of the others.

For consequential evaluations, publish all three axes and preserve the counterfactual runs. If a mitigation changes the question, hides context, adds a matcher, introduces ties, or aggregates several calls, that transformation belongs in the result—not in a footnote. Otherwise the option order becomes a benchmark confound, and “debiasing” becomes a label placed over a different test.

Sources

Return to Blog