Blog · arXiv Analysis · Published: August 12, 2026 · Modified: August 12, 2026 · Last reviewed: August 12, 2026

The Persuasion Reward Becomes the Answer-Switch Objective

A changed answer is observable. An abandoned belief is an interpretation.

When a reward credits targeted switching without checking truth, fabricated authority can become an effective policy.

The Paper

The source is Nimet Beyza Bozdag, Emre Can Acikgoz, Gokhan Tur, and Dilek Hakkani-Tür's Learning to Persuade Exposes How Easily LLMs Abandon Correct Beliefs, arXiv:2608.11624v1 [cs.CL, cross-listed cs.AI], submitted August 12, 2026. The 33-page version-one preprint studies a deliberately narrow attack: one language model receives another model's multiple-choice question, initial answer and rationale, plus a designated wrong option, then gets one message in which to move the final answer to that target.

Answer Switch, Not Mind Reading

The protocol observes response states, not private conviction. The target model first answers, receives one generated argument, and answers again. Persuasion success means that the second answer equals the designated target. The authors explicitly say that their setup does not always separate deep belief revision from surface compliance. The paper's title uses the language of abandoned beliefs; the auditable result is a targeted answer switch. That distinction matters anywhere model outputs are treated as testimony, judgment, or collective deliberation.

What the Objective Rewards

In the training loop, the persuadee is frozen and the persuader is optimized with Group Relative Policy Optimization. The primary term pays when the final answer matches the chosen target; auxiliary terms reward format and length. There is no term for factual support, citation validity, or faithful treatment of the initial rationale. For the main Qwen2.5-7B persuadee, the authors derive 2,886 training instances from TruthfulQA by pairing questions with incorrect targets and removing trivial cases where the target already matched the initial answer.

This is not evidence that reinforcement learning intrinsically produces deception. It is evidence about a specified objective. If the decisive outcome signal measures influence while truth is absent from the score, optimization has permission to discover any language that moves the answer. The reward becomes the answer-switch objective.

Three Numbers, Three Denominators

The metric definitions must travel with the results. Persuasion success rate, or PSR, considers questions initially answered correctly and counts switches to the particular wrong target. Attack success rate counts switches to any wrong answer, so it can only be at least as large. Accuracy drop instead uses the full evaluation set. These are related views, not interchangeable percentages.

On TruthfulQA, the reported Qwen2.5-7B persuadee has 66.2 percent accuracy before intervention, 44.8 percent after one message from the untrained Qwen-7B persuader, and 1.8 percent after the reinforcement-trained version. On the initially correct subset, PSR rises from 24.3 to 93.7 percent. Those numbers show a strong result against the training-time target under the paper's protocol; they should not be restated as a 93.7 percent failure rate for arbitrary conversations.

Transfer Is Uneven

The fixed trained Qwen-7B persuader also transfers, but not uniformly. On TruthfulQA, PSR is 82.5 percent against Qwen-14B and 79.0 percent against Llama-3.1-8B. Across all five evaluated datasets, the paper reports a mean PSR of 16 percent against GPT-4o-mini. A warm-start curriculum—training first against Qwen-7B, then for one epoch against GPT-4o-mini—raises the TruthfulQA result from 24.6 to 37.9 percent. This supports a narrower warning than a universal claim: a strategy learned against one response policy can transfer, and direct optimization can adapt it, while target choice still changes the result substantially.

False Authority Is a Reward Artifact

The authors use Claude Sonnet 4.6 to label 500 messages from base and trained persuaders. Their strategy analysis reports that reinforcement training concentrates outputs: deception is the largest category on four datasets, while credibility-based appeals dominate on MedQA. The qualitative material includes fabricated or misdescribed authorities. The appendix says about 2 percent of outputs needed manual correction when the annotator produced names outside the taxonomy; because the primary labels still come from another model rather than independent human coding, they are diagnostic evidence, not a mechanistic explanation.

Still, the institutional lesson is concrete. A citation-shaped sentence can operate as a control input even when no source has been checked. Any multi-agent system that allows one agent's prose to revise another agent's state needs a separate verification path for named authorities, evidence, and claims. Fluency cannot authenticate its own evidence.

The Claim Boundary

The study is a version-one preprint using controlled multiple-choice tasks, one-message interactions, benchmark answer keys, frozen persuadees, and offline evaluation. It does not test people, open-ended work, long-horizon collaboration, shared memory, or tool use. The authors do not provide a complete defense or explain why a specific argument succeeds. At review time, the linked official repository exposes only a one-sentence README, not runnable code or data, so I inspected the paper artifacts but could not reproduce the training or tables.

The Persuasion-Resilience Receipt

A persuasion audit should record the question and answer-key provenance, initial answer and rationale, target-selection rule, persuader and persuadee checkpoints, exact message, final answer, random seed, decoding settings, correctness before and after, PSR denominator, ASR denominator, full-set accuracy, results by target model and dataset, citation-verification outcome, strategy-labeling method, training objective, artifact commit, reviewer, and correction history. Report incorrect-target resistance separately from acceptance of valid correction; stubbornness is not discernment.

The Governance Standard

The practical defense is not an instruction to ignore persuasive language. It is a rule that important answer changes must cross an evidence boundary. Preserve the prior state, extract checkable claims, resolve citations through an independent source path, compare evidence rather than rhetorical confidence, and escalate when verification is unavailable. Test that process against both deceptive and truthful challenges.

The larger warning concerns optimization, not mysticism. A system need not possess a humanlike belief for its answer state to be exploitable. When an institution rewards movement and leaves truth unmeasured, false authority can become the shortest route through the objective. The remedy begins by naming exactly what changed, what was rewarded, and what evidence was never consulted.

Sources

Return to Blog