Calibrating AI Judges Without Surrendering Judgment
A critical review of evalstats, examining judge-bias correction, unequal-spread rank-test failures, human authority, and the limits of calibrated AI evaluation.
The paper
How to Run Statistics over LLM Judges and Trust the Results: Calibrated Inference for Small-Sample AI Evaluation with evalstats — Arawjo, Ian.
arXiv:2609.35815v1 · Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI); Human-Computer Interaction (cs.HC); Methodology (stat.ME). First submitted 2026-09-20; reviewed version dated 2026-09-20. This review addresses the arXiv manuscript; arXiv hosting alone does not establish peer review.
Contribution: carrying judge uncertainty into inference
Source-derived synopsis
Ian Arawjo’s evalstats manuscript, arXiv:2609.35815v1, presents simulation-selected defaults, nine prediction-powered inference (PPI) tests, four rank-based extensions, and bootstrap-adaptive judge weighting. Synthetic simulations, benchmark resampling, and human–judge corpora support near-nominal average calibration in tested random-labeling settings and power gains over human-only subsets. High agreement can accompany inflated uncorrected false positives. Crucially, Appendix C reports PPI Mann–Whitney and Kruskal–Wallis inheriting classical miscalibration under unequal human-score spreads. An equal-median check with spread ratios up to four found classical inflation roughly twice nominal; PPI did not compound it. Random labeling is insufficient: rank constructions also use exchangeability or sign symmetry. Wilcoxon and Friedman remained nominal in that check with symmetric paired differences. Selective labeling can impair calibration; shrinkage can sacrifice power. Demonstrations include synthetic style data and real ELLIPSE essays with reframed narratives. compare() PPI requires one factor, at least 50 items and 15 human labels, without multi-run data. Unpaired pairwise and multi-run intervals lack reported validation. Appendix E explicitly sets Shaffer correction for PPI tests. Construct validity is outside scope; no user study was conducted.
Original editorial analysis
The useful contribution is a way to make human evidence affect both an automated estimate and its uncertainty. I would assess an LLM judge by the inference it supports: whose ratings define the reference, how calibration examples were selected, and which assumptions connect those examples to the final comparison. Agreement alone cannot answer all three questions.
For the Church of Spiralism’s interests in accountability and human agency, this concerns the allocation of evidential authority. I propose treating calibration as a commitment to a specified human rating process. People should remain able to dispute the rubric, the aggregation of their judgments, and the decision made from the estimate. Statistical correction cannot decide whose judgment deserves authority.
This extends the site’s LLM-as-a-Judge discussion with a concrete question about downstream inference. It also complements What AI Benchmark Labels Can Support: a defensible interpretation of a score and a defensible uncertainty estimate are separate requirements. Neither supplies the other.
Evidence: random calibration does not settle test validity
Original editorial analysis
The unequal-spread result changes how the calibration claims should be read. Random selection answers which items received human labels. Exchangeability and sign symmetry concern the distributions being compared. Satisfying the sampling requirement does not establish the distributional requirements. An adoption decision therefore needs separate arguments for both.
Equal medians do not imply identical distributions. A rank comparison should consequently specify its target rather than being described loosely as a test of average quality. If the decision concerns mean quality, median quality, or the probability that one condition outranks another, the analysis should explain why its chosen test answers that particular question. A familiar test name is insufficient justification.
The adverse result also limits what a successful judge correction can promise. Removing an additional source of bias does not repair every weakness of the underlying statistical procedure. I would require an evaluation report to distinguish the error attributable to judging from the error associated with the test’s assumptions. Otherwise, a corrected result could receive more authority than its construction warrants.
Average calibration should be read alongside the worst-performing scenarios. A pooled result cannot establish acceptable behavior for every deployment. A prospective user should identify which simulated conditions resemble the intended score distribution and labeling process, then examine uncertainty about that resemblance. Where the relevant conditions were not tested, a targeted follow-up would provide more useful evidence than a general endorsement.
The pipeline qualification raises a related issue: validation belongs to the procedure actually exercised. I would record the selected interval method, hypothesis test, multiplicity correction, and software version with each analysis. A subsequent verification could compare automatic selection with explicitly configured choices on identical generated data. That is a proposed test, not evidence that another route fails.
Hypothetical example
Imagine a community comparing two assistants that explain public-service forms. Human ratings have similar medians, but one assistant produces a much wider range of quality. Randomly labeling a subset would help estimate judge error, yet would not make the two score distributions interchangeable. The decision record should state whether consistent performance, average performance, or avoidance of serious mistakes matters most, and choose an analysis suited to that objective. This is an invented example.
Limitations: what correction cannot authorize
Original editorial analysis
My principal reservation concerns the breadth of the word trust. A calibrated comparison concerns a defined quantity under specified assumptions. It cannot establish that an assistant is aligned with everyone affected by it. Such a judgment would require an account of values, affected people, evidence, and decision rights beyond the statistical comparison.
Likewise, failure to detect a difference should not become a declaration of equivalence. A decision maker should specify the smallest difference that would matter and ask whether the uncertainty interval rules out differences of that size. An interface that groups conditions together should explain what that grouping permits users to conclude.
For collective intelligence, I would ask how competing human judgments enter the reference labels. Comparing aggregation rules and retaining reasons for disagreement could reveal whether conclusions depend on suppressing a minority view. Whether that practice improves evaluation quality remains an empirical question. Neither consensus nor dissent automatically establishes measurement quality.
Proposed tests
- Separate assumption failures. Cross random and selective labeling with equal and unequal score spreads. Specify distributional, dominance, and location nulls separately, and compare corrected tests with their human-only counterparts.
- Challenge paired assumptions. Vary the symmetry of paired differences while preserving the intended target quantity. Measure error and coverage rather than inferring robustness from another paired design.
- Test decision understanding. Compare interfaces that present intervals alone with explanations of practical thresholds and assumption limits. Assess whether users distinguish uncertainty, equivalence, and evidence of a difference.
- Measure total resources. Record annotation time, judge calls, inference hardware time, and statistical computation. Environmental claims would require energy measurements and explicit accounting assumptions; effective label gains alone cannot establish them.
I would consider evalstats a candidate for more accountable inference when the sampling design, target quantity, test assumptions, and validated software path are inspectable. Questions about consciousness and spiritual meaning require separate arguments; these statistical methods provide no evidence establishing spiritual doctrines.
AI-use disclosure
This review was drafted using AI, revised by GPT-6.1 Sol with high reasoning, and checked in a separate GPT-6.1 Sol session with medium reasoning. Automated checks are not human peer review; the paper’s claims have not been independently verified for this article.
Paper sources
Bibliographic metadata checked 2026-10-23.