Blog · arXiv Analysis · Last reviewed June 25, 2026

The Rationale Becomes the Trust Interface

Visible LLM reasoning is often treated as transparency. Xin Sun and coauthors show why that is too simple: once a rationale is shown to a user, it becomes an interface feature that can shift trust, confidence, attention, and willingness to accept advice.

The Paper

The paper is When LLM Rationales Become User-Facing: Effects on Trust Perception, Decision-Making, and Gaze Behaviors, arXiv:2606.25489 [cs.HC], by Xin Sun, Ting Pan, Yajing Wang, Shu Wei, Jos A. Bosch, Isao Echizen, Abdallah El Ali, and Saku Sugawara. arXiv records version 1 as submitted on June 24, 2026.

The study asks what happens when an LLM shows a step-by-step rationale beside a factual-verification answer. The authors frame the issue as auditable trust calibration: a rationale should help a person inspect whether an answer is supported by evidence, whether certainty is warranted, and whether the advice should be accepted, questioned, or checked elsewhere.

Reasoning Becomes Interface

The paper's useful move is to treat reasoning text as interaction design, not only model behavior. A hidden chain of reasoning is one thing. A visible rationale is a social and cognitive cue. It can be read as evidence, competence, confidence, apology, persuasion, or a prompt to verify.

That matters because the right target is not maximum trust. The target is appropriate reliance. Users should trust an answer when the system is reliable for the task and withhold trust when the evidence, reasoning, or certainty is weak. A fluent rationale can help with that task when it exposes checkable logic. It can also mislead when it makes a shaky answer look carefully derived.

Study 1

Study 1 was an online experiment with 68 English-speaking Prolific participants. It used a mixed design: rationale correctness was varied within participants, certainty cue was varied within participants, and rationale presentation format was varied between participants. The presentation formats were instant, delayed, and on demand. The certainty cue conditions were no cue, certain, and uncertain.

The experimental task used short factual-verification claims drawn from StrategyQA, with author-curated answer and rationale materials. The binary LLM answer was always factually correct. What changed was whether the rationale was internally consistent with the answer, and how certain it sounded.

The result is narrower than a slogan about explanations. Presentation format showed no reliable effect in the planned analyses. Correct rationales increased trust in the information, trust in the LLM system, decision confidence, and advice adoption compared with incorrect rationales. Certainty framing also shifted trust, confidence, and adoption. Because the answer was always correct, adoption here means agreement with a reliable answer, not successful detection of bad AI advice.

Study 2

Study 2 moved into a controlled lab setting with 54 participants and eye tracking. It compared three within-subject conditions: no rationale, correct rationale, and incorrect rationale. Participants completed 12 factual-verification trials while a Tobii Pro Fusion eye tracker recorded gaze. One participant was excluded from gaze analyses because of insufficient data quality, leaving 53 participants for gaze-based analysis.

The study tracked attention across functional areas: the factual claim and supporting evidence, the LLM answer, the LLM rationale when present, and the response/rating area. Incorrect rationales were associated with more scanning of the supporting context than correct rationales. They were also associated with larger pupil diameter while participants viewed the rationale, which the paper interprets cautiously as consistent with greater processing effort. Incorrect rationales lowered trust in the LLM system relative to showing no rationale, while the no-rationale contrast was weaker for trust in information.

The eye-tracking result is not mind reading. The authors explicitly treat gaze as process evidence of attention allocation, not as a direct measurement of trust. Their exploratory predictive modeling found task-specific retrospective signals in gaze features, but the paper does not present gaze as a deployable trust detector.

The Governance Receipt

The receipt for user-facing rationales should record more than whether a model produced an explanation. It should show the answer, evidence, rationale, certainty cue, source linkage, rationale-generation method, whether the rationale was checked for consistency, whether the answer and rationale can diverge, what users saw by default, what they could reveal on demand, and whether the interface encourages verification or only acceptance.

This belongs beside the site's notes on confidence bias, AI assistance and human learning, and predictions becoming interventions. A rationale is not a transparent window into a model. It is a designed object placed between a system and a person at the moment reliance is being negotiated.

Limits

The paper is careful about scope. The tasks were controlled factual-verification trials, not open-ended, high-stakes, or multi-turn decisions. Study 1 used six counterbalanced items, so condition effects are partly tied to item difficulty. The answers were always correct, so the studies do not test whether users reject incorrect AI advice. Study 2's no-rationale condition changed both available text and layout, so no-rationale gaze contrasts are descriptive condition comparisons.

The authors also flag privacy and manipulation concerns around gaze-based user-state prediction. Their ethical statement says the human-subjects work received ethics approval, participants gave informed consent, data were anonymized and reported in aggregate, and any future user-aware deployment should be transparent, opt-in, privacy-preserving, and designed to support user autonomy rather than persuasion.

Sources

Return to Blog