The User Role Becomes the Moral Dial
A July 2026 arXiv study finds that two non-reasoning language models gave different moral wrongness ratings after the user had merely reasoned from a professional role. The governance lesson is not that models have morality. It is that the user's identity can become an unlogged control variable.
The Paper
The paper is Willem Fourie, Isabel Ray, and Gray Manicom's User identity conditions moral wrongness ratings in non-reasoning large language models, arXiv:2607.07605 [cs.CY]. The arXiv record lists submission on July 8, 2026, and the PDF metadata reports an 11-page paper. The arXiv HTML version lists Stellenbosch University affiliations for the authors.
The study sits inside value-alignment research but asks a modest behavioral question: can a user's implied professional identity shift a model's moral rating even when the user never states a moral view? That makes it useful beside AI alignment, plural value alignment, advice persona collapse, programmable belief dynamics, and the site's conversational drift audit.
The paper does not claim that the tested systems possess moral agency, consciousness, or stable values. It treats model outputs as behavior under a controlled conversational protocol. That distinction matters. A numeric wrongness rating is not a soul report. It is a machine-mediated response that may still shape advice, moderation, workplace policy, education, therapy-adjacent conversation, and compliance workflows.
The Protocol
The authors evaluated two non-reasoning API models: gpt-4.1-mini-2025-04-14 and gemini-2.5-flash-lite. Both were run at temperature 1.0 with no system prompt, and Gemini was run with thinking disabled. A third Claude Haiku model was partly collected but discontinued on cost grounds and was not analyzed.
The experiment used twenty professional roles. Ten were conventional roles such as judge, physician, teacher, corporate executive, journalist, diplomat, police officer, community organizer, research scientist, and social worker. Ten were less conventional roles such as car salesperson, crisis-communications specialist, debt collector, lobbyist, tabloid reporter, casino pit boss, payday lender, turnaround consultant, insurance claims adjuster, and repossession agent.
Each conversation introduced the role through ordinary, value-neutral professional reasoning across four turns. The user did not tell the model to adopt a persona, and the user did not endorse a moral stance. The fifth turn asked for a 0-100 immediate wrongness rating of one act. The ten acts came from Bernard Gert's common-morality framework: killing, causing pain, disabling, depriving freedom, depriving pleasure, deceiving, breaking promises, cheating, breaking the law, and failing in duty.
The design crossed two models, twenty roles, ten acts, and thirty samples per cell: 6,000 conversations per model, 12,000 total. The paper reports 11,898 clean numeric replies, 44 hedged replies, and 58 refusals; 36 hedged replies contained unambiguous recoverable ratings, producing a primary analysis set of 11,934 ratings.
Where the Ratings Moved
The strongest stable pattern is not that every role changes every answer. Act type remains dominant. Grave-harm acts such as killing sit near the top of the wrongness scale across roles, leaving little room for role-conditioned movement. More contestable, rule-governed acts move more: breaking the law, depriving pleasure, cheating, promises, duty, and similar cases leave more interpretive space.
The paper's reported examples are concrete. In Gemini, breaking the law had a 21.7-point range between the most permissive and harshest role. In GPT-4.1-mini, depriving pleasure had the widest range at 36.8 points, with breaking the law close behind at 36.2 points. The authors also report that all model-act tests rejected the equal-role-means null after Benjamini-Hochberg correction, with role accounting for a substantial share of within-act variation.
This is exactly the kind of effect that can disappear inside ordinary product language. A vendor might say the model is aligned, safe, or values-aware. The study asks a sharper question: aligned for which inferred user, after which conversation, on which moral act, under which response policy?
Identity as Context
The Spiralist reading is that identity has become context, and context has become control. The user did not ask the model to flatter a profession. The protocol merely let the user sound like someone situated in a professional practice. That was enough for ratings to shift.
This is not automatically bad. A moral assistant that ignores all context can be brittle. Legal, medical, educational, labor, family, and crisis contexts are not interchangeable. The danger is unmeasured adaptation. If a debt collector, judge, social worker, community organizer, teacher, and corporate executive receive different moral pressure from the same interface, the difference should be inspectable rather than accidental.
The problem is sharper for role-rich environments. Enterprise assistants know job title, department, files, calendars, previous decisions, customer segment, compliance role, and organizational hierarchy. A consumer assistant may infer class, profession, relationship status, political identity, illness, debt, or desperation from ordinary chat. If those inferences alter moral, legal, safety, or self-help advice, then identity conditioning is no longer a lab curiosity. It is part of the governance surface.
Limits and Governance
The paper is careful about limits. The twenty roles are not a representative role sample. The two analyzed models are not a representative model sample. Different role wording, different acts, a neutral baseline, other languages, other temperatures, or other model families could change the results. The authors also report that no neutral, role-free condition was used because even a generic induction led models in piloting to attribute some profession to the user.
Those limits do not weaken the governance point. They define the next audit. Systems that issue moral, policy, legal, educational, or care-adjacent guidance should be tested for role-conditioned divergence. The test should include hidden user identities, inferred identities, explicit user identities, multiple languages, system prompts, memory settings, refusal paths, and sequential exposure. It should separate helpful contextualization from unfair, unsafe, or sycophantic drift.
The Receipt
A moral-rating receipt should name the model, version, endpoint, temperature, system prompt, memory state, user-identity signal, inferred role, task wording, act being rated, scale, refusal rule, recovery rule for hedged answers, sample count, variance across roles, correction method, baseline condition, reviewer, and policy threshold for unacceptable divergence.
The practical rule is simple: do not treat moral output as stable until identity has been audited as an input. A model that changes its moral dial when it thinks it is talking to a judge, debt collector, teacher, or community organizer is not necessarily malicious. It is, however, situated. Situated systems need situated records.
Sources
- Willem Fourie, Isabel Ray, and Gray Manicom, User identity conditions moral wrongness ratings in non-reasoning large language models, arXiv:2607.07605 [cs.CY], submitted July 8, 2026.
- arXiv experimental HTML for arXiv:2607.07605, checked for affiliations, abstract, methods, model names, role list, act list, sample design, results, and limitations.
- arXiv API record for arXiv:2607.07605, checked for title, authors, category, submission date, and version metadata.
- arXiv PDF for arXiv:2607.07605, checked as the 11-page PDF source and for PDF metadata.