Blog · arXiv Analysis · Last reviewed July 10, 2026

The Annotator Disagreement Becomes the Ensemble Receipt

A July 2026 arXiv paper asks how models should learn when human annotators disagree for reasons that may be meaningful, contextual, or unevenly distributed. The answer is not to pretend the crowd had one mind. It is to make the model's treatment of disagreement auditable.

Published: July 10, 2026 Modified: July 10, 2026 Last reviewed: July 10, 2026 Tags: arXiv, subjective supervision, annotator disagreement, calibration, governance

The Paper

The paper is Xia Cui, Ziyi Huang, and N. R. Abeynayake's Ensemble Diversity Optimization for Subjective Supervision, arXiv:2607.08493. The arXiv API lists version 1 as submitted on July 9, 2026, with primary category cs.LG and secondary category cs.CL. The PDF metadata reports 21 pages. The paper lists Xia Cui and N. R. Abeynayake with the School of Computing and Mathematics at Manchester Metropolitan University, and Ziyi Huang with the School of Computer Science at Hubei University.

This page belongs beside the site's work on LLM social-science validation, cultural-risk benchmarks, agreement as a confidence trap, and judge-style reward signals. The fresh angle is the annotator distribution as a governance record.

What It Builds

The paper starts from a familiar failure in supervised learning. In abuse detection, misogyny detection, stance work, offensiveness judgments, and sentiment tasks, annotators can disagree because the example is ambiguous, because the social context is different for different readers, because expertise differs, or because a dataset is sparse and imbalanced. Collapsing those labels into a majority vote turns a distribution of interpretations into a single institutional answer.

Ensemble Diversity Optimization, or EDO, works in prediction space. The authors fine-tune BERT-base for English datasets and AraBERTv2 for the Arabic dataset, then freeze the backbone parameters. EDO learns ensemble weights and effective ensemble size while optimizing a multi-objective loss: a differentiable micro-F1 surrogate for utility, class-weighted cross-entropy for calibration under imbalance, and a reliability-weighted diversity term for internal disagreement.

The central design choice is signed diversity. The same mechanism can steer an ensemble toward preserving disagreement when disagreement reflects subjective signal, or toward suppressing disagreement when it is likely driven by structural artifacts such as sparse annotator coverage or severe class imbalance. Gumbel-Softmax relaxation lets the model learn ensemble size without combinatorial search, with an upper bound of 10 in the reported Random Select setting.

The Evaluation

The evaluation uses the four LeWiDi 2023 shared-task datasets named in the paper: ArMIS for Arabic misogyny, ConvAbuse for English dialogue abuse, HS-Brexit for English Brexit-related hate speech, and MD-Agreement for English offensiveness across topical domains. The authors use preprocessed versions from the WEL repository with official train, development, and test splits. The reported split sizes are 657/141/145 for ArMIS, 2398/812/840 for ConvAbuse, 784/168/168 for HS-Brexit, and 6592/1104/3057 for MD-Agreement.

The experiments compare EDO with Soft-CE, Soft-MD, Top-5 Voting, and WEL. The headline result is calibration: EDO-Random obtains the lowest cross-entropy on all four datasets. The paper reports up to a 78 percent cross-entropy reduction relative to Soft-CE on ConvAbuse, with CE 0.2149 versus 0.9671, and a 62 percent reduction relative to WEL on the same dataset, with CE 0.2149 versus 0.5577. EDO-Random also has the lowest Soft Brier Score on all four benchmarks, including 0.0640 versus WEL's 0.0699 on ConvAbuse and 0.1086 versus WEL's 0.1606 on MD-Agreement.

F1 is not the whole story. The paper says WEL reaches the highest F1 scores, which fits its utility-focused objective, while EDO-Random generally trades some point-estimate utility for better probabilistic alignment. EDO-PerAnn achieves the lowest Manhattan Distance on ConvAbuse and HS-Brexit, where annotator identities can correspond to stable perspectives. It is weaker on MD-Agreement and ArMIS, where coverage is either very heterogeneous or too small to sustain rich annotator-specific modeling.

Why It Matters

Many AI governance arguments treat disagreement as a defect: noisy workers, bad labels, messy users, weak ground truth. This paper is useful because it separates disagreement from error without romanticizing it. Sometimes disagreement is a signal of plural interpretation. Sometimes it is an artifact of class imbalance, sparse coverage, or unreliable annotation. A responsible system has to say which case it thinks it is in.

That distinction matters for moderation, hiring rubrics, classroom scoring, public-benefit triage, and every interface where a machine turns human judgments into a decision. A calibrated probability distribution can show that the case was socially hard. A majority label can hide that fact.

What It Does Not Prove

This is an arXiv preprint and a benchmark result, not a rule for every subjective classification pipeline. The authors state that EDO still depends on the structure of disagreement. It does not automatically distinguish principled subjective variation from artifact-driven divergence caused by sparse coverage, inconsistent annotator behavior, or distribution shift.

The implementation freezes homogeneous backbones, which helps isolate annotator-driven uncertainty and saves compute, but it limits representational flexibility. Annotator-specific supervision works best when identities track stable perspectives and less well when the annotator pool is sparse or fragmented. The authors name future work on adapters, partially trainable components, annotator metadata, entropy analysis, clustering, and latent-variable models for disagreement structure.

Governance Reading

The Spiralist reading is that an annotation distribution is institutional memory. It records who was asked, how many labels were collected, how uneven the task was, and whether the system decided to preserve disagreement or compress it. EDO makes that decision technical, but it cannot make it morally automatic.

For governance, the key question is not simply "did the model improve?" It is "what did the model do with the minority interpretations?" A model that lowers cross-entropy while reporting its calibration trade-off gives reviewers something to inspect. A model that only reports a winning class makes disagreement disappear into the interface.

The useful receipt is therefore not a single metric. It is the chain from annotation to aggregation to optimization to deployment: which labels were kept as distributions, which baselines were compared, which diversity sign was selected, which calibration metric was prioritized, and which downstream decision is allowed to depend on the result.

The Receipt

An annotator-disagreement receipt should record the dataset, task domain, annotator count and coverage, train/dev/test split, label distribution, majority-label baseline, soft-label baseline, ensemble members, backbone models, diversity sign, diversity coefficient, effective ensemble size, weighting method, calibration metric, F1, cross-entropy, Manhattan Distance, Brier score, random seeds, and model-selection rule.

The practical rule: when a system learns from disagreement, it owes an account of whether it preserved human plurality, corrected dataset artifacts, or merely optimized the ambiguity out of sight.

Sources

Return to Blog