When Novices Must Judge AI Without Feedback
A critical review of a study on novice reliance on AI suggestions, examining the evidence for explanations, self-assessment, and meaningful oversight.
The paper
Novice Reliance Calibration in AI-Assisted Decision Making: The Role of Explanations and Self-Assessment — Kang, Eun Jeong; Peter; Duan; Mishra, Swati.
arXiv:2610.07800v1 · Human-Computer Interaction (cs.HC) ; Artificial Intelligence (cs.AI). First submitted 2026-10-06; reviewed version dated 2026-10-06. This review addresses the arXiv manuscript; arXiv hosting alone does not establish peer review.
Contribution
Kang, Duan, and Mishra study how novices use AI suggestions when they receive no immediate correctness feedback. In a randomized, four-condition study, 110 analyzed participants annotated clinical concepts in discharge summaries. Two conditions provided AI highlights, with one also offering a medical dictionary and model confidence scores. The authors recorded acceptance, editing, and deletion of suggestions and analyzed reliance using a Brier-like score and Bayesian mixed-effects models. Across the AI conditions, participants accepted 75% of suggestions. The explanation package did not show a credible improvement in the main calibration score; higher self-reported understanding was associated with more editing or rejection of incorrect suggestions. The paper also proposes interface interventions, including prompts based on users’ reliance patterns. Those interventions were not tested in this experiment. The review addresses arXiv:2610.07800v1, submitted October 6, 2026.
Original analysis: The paper gives a concrete meaning to a difficult governance question: can a person distinguish useful assistance from a plausible error when the system does not tell them whether they were right? That question matters for human agency because an available edit button is only one part of effective judgment. The person also needs a basis for deciding when to use it. This study examines that narrower ability in an annotation interface; it does not establish whether people have authority to halt a deployed system or contest an institution’s decision.
The distinction complements the site’s discussion of human oversight. Here, the useful contribution is an observable comparison between actions and the correctness of suggestions. The experiment makes it possible to ask whether the person’s intervention tracks the quality of the advice, rather than treating any human interaction as evidence of oversight.
Evidence and interpretation
Original analysis: The explanation comparison is informative but specific. The AI-only group had 23 analyzed participants and the AI-plus-explanation group had 28. The latter received a package consisting of a dictionary and confidence scores, so the comparison cannot isolate the effect of either component. The estimated explanation effect on the reliance score was −0.02, with a 95% credible interval from −0.05 to 0.02. That interval supports caution about claiming an improvement; it does not prove every possible explanation is ineffective. The paper also reports a stronger correlation between reliance and correctness in the explanation condition, even though its main score did not show a credible improvement. Those measures capture different features of behavior and should not be collapsed into one verdict.
The authors report more over-reliance than under-reliance overall. Their estimated increase in over-reliance over the session had a 95% credible interval of −0.01 to 0.09, while the overall calibration score’s increase was not statistically clear. The abstract’s language of drift is therefore stronger than the reported interval alone establishes. There was substantial variation between participants’ trajectories. The results are a reason to investigate changing reliance, not a precise demonstration that explanations caused a steady deterioration.
Self-assessment deserves similar care. The paper reports that higher pre-task self-reported understanding was associated with rejecting or editing incorrect suggestions and adding missed entities. Confidence alone did not show a credible improvement in appropriate acceptance. These are associations, not evidence that increasing a person’s feeling of understanding would cause better decisions. The paper also reports an interaction-model coefficient of −0.03 with an interval of −0.14 to −0.08; the point estimate lies outside that interval. This apparent reporting inconsistency makes that particular estimate difficult to interpret without clarification from the authors.
Original analysis: The reliance score is useful because it includes edits as well as acceptance and deletion. Its design also makes a substantive choice: every edit receives a reliance value of 0.5, regardless of whether the edit fixes an error or introduces one. A suggestion’s correctness is assessed against reference annotations, while the score focuses on responses to suggestions. The authors separately analyze additions of missed entities, but the main score alone cannot describe the quality of the complete final annotation. A low score is therefore evidence about this defined alignment between actions and suggested items, not a complete measure of task success or independent judgment.
Limitations and proposed tests
Original analysis: The population was recruited through Prolific and reported little prior healthcare knowledge. The task involved text annotation, not clinical diagnosis or treatment. Its short sessions, category definitions, interface defaults, and model error patterns all limit transfer to other settings. The paper reports 58.79% correctness for suggestions on the tested documents under its matching criteria; that figure is distinct from the model performance figures reported elsewhere in the method. The source also describes up to 42 possible tasks per participant but reports a maximum of 102 tasks completed. Clarification of that discrepancy would help readers assess exposure and session trajectories.
Proposed test: A follow-up could randomize dictionary access and confidence scores separately, then compare them with AI-only support and a condition that provides checked feedback. It could measure final annotation quality, responses to correct and incorrect suggestions, and missed entities. Reporting uncertainty for each outcome would help distinguish better error detection from a general tendency to edit more often.
Hypothetical example: An annotator repeatedly accepts highlighted numbers in a lab-value category. A prompt could ask the annotator to check whether a number is actually a lab value in context. The prompt might encourage useful review, or it might interrupt work without improving accuracy. The paper proposes behavior-triggered support; it does not establish that this particular prompt works.
Proposed test: Researchers could compare such prompts against no prompt and measure both corrected errors and unnecessary changes, alongside time spent and user experience. If behavioral monitoring is used, a governance evaluation should also ask who can see the logs, how long they are retained, and whether they are used to support learning or judge workers. These are proposed safeguards and research questions, not measured outcomes of the study.
The findings also give the broader account of automation bias a specific boundary: over-reliance appeared in this novice annotation task, but the study does not show that every human-AI workflow produces the same pattern. For a community concerned with accountability, its most useful lesson is to ask for evidence about what people can actually detect and correct, while keeping the paper’s exploratory design proposals separate from validated remedies.
AI-use disclosure
This review was drafted by GPT-6.1 Terra with high reasoning and checked in a separate GPT-6.1 Sol session with medium reasoning. Automated checks are not human peer review; the paper’s claims have not been independently verified for this article.
Paper sources
Bibliographic metadata checked 2026-10-27.