The Preference Debate Becomes the Constitution

A Label Is Not a Reason

Preference data is often treated as if it contained a complete moral signal. One response wins, the other loses, and the downstream system learns from that ordering. But the label usually omits the argument. A person may prefer an answer because it is more specific, less sycophantic, more careful with uncertainty, more vivid, less manipulative, or simply more useful in context. The binary label compresses all of that into a single bit of institutional memory.

That compression matters when preference data becomes alignment infrastructure. If the reason disappears, later auditors can see only the verdict, not the standard that produced it. The useful question is therefore not whether a model can imitate a preference distribution. It is whether the system can recover enough of the reasons to make the standard visible, contestable, and revisable.

The Paper Frame

The source is Kevin Kingslin, Anish Natekar, Ashutosh Ranjan, Vivek Srivastava, Savita Bhat, and Shirish Karande's Democratic ICAI: Debating Our Way to Steering Principles from Preferences, arXiv:2606.28294 [cs.LG], submitted June 26, 2026. The arXiv record lists it as accepted to the ICLR 2026 HCAIR Workshop.

The paper extends Inverse Constitutional AI, or ICAI. ICAI tries to infer natural-language principles from preference comparisons. Democratic ICAI, shortened by the authors to DICAI, changes the extraction step: instead of asking for one explanation, it asks multiple expert personas to generate rationales, subjects those rationales to structured debate, and then distills the settled criteria into a compact constitution.

From Pair to Parliament

The pipeline starts with paired samples and human preference labels. For each pair, the authors generate rationales through a committee of three task-specific personas. Each persona uses a different reasoning strategy: chain-of-thought, self-refine, or self-consistency. The goal is to surface different plausible reasons before anything is summarized.

The next stage is adversarial debate, implemented with AutoGen in the reported experiments. Personas challenge and refine the candidate rationales while a judge consolidates overlapping criteria. After three debate rounds, the resulting principles are embedded with OpenAI's text-embedding-3-small model, grouped with K-Means clustering, and abstracted into a human-readable constitution. That constitution is then used in two ways: as guidance for an LLM judge and as features for a decision-tree judge.

Creative Preference Tests

The paper tests the method on creative-preference tasks rather than simple factual correctness. It uses MuCE-Pref for short-form creative tasks and LiTBench for long-form stories. For MuCE-Pref, the authors sample 500 training pairs per task to induce task-specific constitutions and evaluate on official test splits. For LiTBench, they use 1,000 pairs for constitution induction and 2,000 held-out pairs for evaluation.

The reported MuCE categories include Alternate Uses of Objects, Consequences, Design Solutions, Experiment Design, Hypothesis Generation, Metaphors, Real-Life Creative Problem Solving, Research Questions, and Short Stories. This matters because creative judgment is not one criterion in disguise. A good metaphor, a good design solution, and a good research question can all be better for different reasons.

What Improved

Using GPT-4o as the LLM judge, Table 1 reports average preference accuracy of 75.10% for DICAI, compared with 67.31% for ICAI and 55.12% for AutoRubric. With GPT-5 as judge, Table 2 reports 75.99% for DICAI, 71.16% for ICAI, and 57.59% for AutoRubric. The authors also state that DICAI has the lowest standard deviation across tasks under both judge models.

The gains are not uniform triumphs over every deliberative prompting baseline on every task. The paper says this directly. The stronger claim is narrower: across these creative-preference tasks, debate-derived constitutions reconstruct the preference structure better on average than the single-pass ICAI and AutoRubric baselines, and an independent decision-tree judge supports the direction of the improvement.

Governance Reading

The governance value is not that a constitution is automatically right. The value is that the evaluative standard becomes inspectable. A reward model can bury a preference pattern inside weights. A DICAI-style constitution produces a list of candidate principles that can be read, criticized, edited, retired, versioned, and compared against policy.

That makes the method relevant beyond creative writing. Any institution using preference data to tune model behavior faces a documentation problem. What values were inferred? Which trade-offs were elevated? Which stakeholder perspectives disappeared? A debate-derived constitution is not an answer to those questions, but it is a better audit object than a scalar reward alone.

Limits and Bias Risk

The limitation section is the part to keep close. The authors say explicit ground-truth principles behind human judgments are often unavailable, so evaluation relies on proxy measures rather than direct validation against human intent. They also warn that preference data can encode systematic biases, which can propagate into induced principles and downstream behavior.

The paper reports a bias and spurious-criteria audit using Qwen2.5-32B as an external auditor and says it found no high-severity flags for the DICAI constitutions. That is useful, but it is not a license to skip human review. The ethical appendix recommends treating generated constitutions as editable artifacts rather than fixed outputs. That is the right governance posture.

Audit Receipt

The audit-grade sentence is: Kingslin, Natekar, Ranjan, Srivastava, Bhat, and Karande present Democratic ICAI, a method that derives steering principles from preference data through three-persona rationale generation, structured debate, clustering, and abstraction, then evaluates those constitutions on MuCE-Pref and LiTBench with LLM and decision-tree judges.

The receipt is: preference governance should preserve the reasons behind the labels, expose the induced principles, record the judge and dataset path, and keep the resulting constitution editable under human oversight.

Sources

• Kevin Kingslin, Anish Natekar, Ashutosh Ranjan, Vivek Srivastava, Savita Bhat, and Shirish Karande, Democratic ICAI: Debating Our Way to Steering Principles from Preferences, arXiv:2606.28294 [cs.LG], submitted June 26, 2026.
• Primary versions checked: experimental HTML and PDF.
• Related pages: The LLM Facilitator Becomes the Steering Wheel, The Policy Table Becomes the Participation Filter, The Aligned Crowd Becomes the Market Monoculture, The Synthetic Respondent Becomes the Public, and The Model Constitution Becomes the GPAI Code.

Return to Blog