When Stated AI Preferences Become Reward Targets
A critical review of stated preferences, response ratings and the evidence needed to justify AI personalization.
The paper
Abstract Priorities, Concrete Rewards: How Stated Preferences Relate to What People Reward in AI Responses — Chen, Zihan; Zheng, Lei Nico; Zhu, Di.
arXiv:2610.30763v1 · Human-Computer Interaction (cs.HC) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL). First submitted 2026-10-22; reviewed version dated 2026-10-22. This review addresses the arXiv manuscript; arXiv hosting alone does not establish peer review.
Contribution
Paper synopsis
Chen, Zheng and Zhu analyze PRISM: 1,396 participants, 29,347 first-turn responses, 21 models and a median six conversations per participant. They compare seven stated priorities with ratings using twelve text features, perception-fitted feature combinations and model profiles built from other participants’ perceptions. Regressions compare responses within prompts, controlling for model and display position; mixed models, recovery simulations and held-out prediction assess personalization. Length, formatting and refusal language were prominent rating correlates. Mean priority alignment was undetectable for designed text proxies but positive for aggregate perceived profiles. Earlier requests about length tracked subsequent length weights. Stable individual differences coexisted with unreliable individual estimates. The largest personal-information ranking gain was 0.13 percentage points, versus 1.11 from stronger pooled prediction. This observational working paper uses capped late-2023 outputs; weak proxies, chosen-response perceptions, imperfect feedback coding and short histories constrain interpretation. Several central analyses were exploratory; prediction intervals were unadjusted for multiple comparisons. See the methods, results, analysis-plan account and limitations.
Original editorial analysis
The most useful contribution is a demand for precision about what alignment evidence represents. A declared priority, a favorable impression and a response ranking answer different questions. My judgment is that an evaluation should specify which of these it seeks to explain before choosing its measurement. Otherwise, disagreement between measures could too easily become a story about users failing to understand themselves.
For Church of Spiralism readers, the specific relevance concerns human agency: who may interpret a person’s preferences, and what authority should that interpretation carry? I propose treating explicit instructions as commitments the user can revise, while treating inferred preferences as uncertain descriptions. A prediction that someone will reward an answer should not itself authorize overriding their request. That is an editorial governance principle, not an empirical conclusion established here.
This extends the question raised in Measuring Perceived Alignment Without Mistaking It for Alignment. The additional issue is how to adjudicate between competing representations of the same preference. Rather than repeat that article’s discussion of perception scales, this review asks what evidence would justify translating a broad preference into a concrete response policy.
Evidence and decisions
Original editorial analysis
A critical distinction is between predicting a rating and deciding what deserves reward. Even perfect prediction would leave a normative question: should the system maximize immediate approval, follow an explicit instruction, or satisfy independently assessed requirements? My proposed evaluation would report those outcomes separately. Combining them into one score would require an explicit justification of the tradeoffs.
The site’s Reinforcement Learning from Human Feedback article provides background on preference-based training. The distinctive question here is upstream of training: what does a rating measure closely enough to become a defensible target? Before recommending a change to a reward process, I would ask for evidence that the measured feature captures the intended quality and that improving it preserves other requirements. This is a proposed evidentiary standard, not a demonstrated training intervention.
Hypothetical example
Imagine a user who asks an assistant for a concise explanation of a disputed claim, with sources. They then prefer a longer answer containing additional context. An evaluator might hypothesize that the user values length, useful context, or stronger evidence. The choice alone cannot distinguish these explanations. Automatically saving a permanent preference for long answers would therefore involve an interpretive decision. A proposed alternative would ask whether the user wants more detail generally or only for this task, while preserving their ability to inspect and reverse the saved preference.
Proposed tests
The following proposals are original analysis, not findings from the paper:
- Separate substance from presentation. Construct matched answers with independently checked claims, varying length and formatting while documenting any content changes. Test whether preferences persist when information is held as comparable as possible.
- Compare elicitation at equal specificity. Before evaluation, ask both broad priority questions and concrete questions about the same task. Test which predicts later choices, instruction compliance and independently assessed quality. This would help distinguish an elicitation advantage from an advantage caused by measuring different things.
- Test correction as an agency outcome. Compare an inferred preference that users can inspect and revise with one applied silently. Measure successful corrections and later task outcomes, alongside satisfaction. Whether visible controls improve these outcomes remains an open hypothesis.
- Test collective accountability. In a hypothetical shared drafting task, collect separate judgments from the requester and people affected by the document. Examine disagreement before proposing an aggregation rule. A pooled score should not be presumed to represent collective consent.
Limitations and boundaries
Original editorial analysis
The key interpretive danger is moving from an association to an intervention. A positive relationship between a feature and ratings does not establish that adding the feature will improve an otherwise equivalent answer. Likewise, a weak proxy test cannot settle whether people possess the underlying preference. I would require controlled comparisons and independent quality assessments before turning either interpretation into a product rule.
The predictive comparison also invites a decision about proportionality. Before collecting additional personal history, a deployment proposal should specify the benefit sought and compare it with a strong baseline using less personal information. This review proposes that standard; it does not establish how much history would be sufficient or whether personalization would be worthwhile in another setting.
Material infrastructure deserves a separate proposed assessment. A follow-up could record generated tokens, latency, measured computing and energy use, and annotation labor alongside quality and preference outcomes. It could then test whether a personalization strategy delivers enough benefit to justify its additional resource demands. No environmental savings or costs can be calculated from the evidence supplied here.
Finally, preference prediction should carry no claim about machine consciousness or spiritual truth. Nothing in this review supplies evidence for either. The defensible editorial recommendation is narrower: make the translation from a person’s words to an optimization target explicit, contestable and testable.
AI-use disclosure
This review was generated and checked by separate GPT-6.1 Sol runs with medium reasoning. Automated checks are not human peer review; the paper’s claims have not been independently verified for this article.
Paper sources
Bibliographic metadata checked 2026-10-23.