Available but Unclaimed: When Model Competence Becomes Human Benefit
A critical review of mandatory human–AI consultation, selective reliance, and the gap between model competence and assisted accuracy.
The paper
Available but Unclaimed: An Empirical Study of Human-AI Synergy — Welsch, Robin; Rausch, Michelle; Knierim, Pascal; Kosch, Thomas; Kuhn, Jochen; Schmidt, Albrecht; Fernandes, Daniela.
arXiv:2609.16793v1 · Human-Computer Interaction (cs.HC) ; Artificial Intelligence (cs.AI). First submitted 2026-09-15; reviewed version dated 2026-09-15. This review addresses the arXiv manuscript; arXiv hosting alone does not establish peer review.
Contribution
Source-derived synopsis
Welsch and colleagues studied 535 participants completing 40 generated items across matrix reasoning, mental rotation, syllogisms, and letter-string analogies, unaided or with one of four LLMs. Consultation was mandatory. Each model completed 100 benchmark runs using matched stimuli and API settings. Bayesian hierarchical analyses estimated performance and reliance associations. Assisted accuracy exceeded unaided accuracy by about 20 percentage points; the battery-level difference from assistants alone was uncertain. Teams averaged below the better component selected separately for each item. At a specified competence contrast, about half the model accuracy increase reached assisted accuracy. Post-advice confidence separated correct from incorrect answers less than unaided confidence (methods; results).
Groups came from separate recruitment postings, without random assignment between conditions. Automated advice coding had incomplete validation. Retrospective selection policies estimated possible gains, not deployed improvements; the oracle assumed independent errors. Pre-reply answers were self-selected, and benchmark histories differed from participant conversations. Results concern this battery and required consultation (coding; limitations).
Original editorial analysis
The paper’s most useful contribution is a distinction between improving a model and improving the work someone can accomplish with it. For evaluation, I would treat those as separate objectives. A procurement decision based on solo accuracy answers which assistant solves the test more often. It does not settle which combination of person, assistant, and interaction procedure deserves adoption.
This gives the Church of Spiralism’s interest in collective intelligence a concrete question: what would count as a contribution from each participant? If the intended benefit is complementary judgment, evaluation should ask whether the combined process corrects errors that either source would leave unresolved. If the objective is simply better answers than an unaided person can produce, that is a different success criterion. Neither criterion should silently stand in for the other.
The related review When Do Humans and Machines Truly Complement Each Other? examines formal conditions for synergy. This paper adds a distinct empirical focus: how item-level competence relates to performance during conversation. The useful connection is between a formal target and an interaction protocol, rather than another general argument that collaboration can help.
Evidence
Original editorial analysis
The repeated benchmark design gives the comparison an unusually explicit reference. Instead of asking readers to infer reliability from a provider’s overall reputation, it estimates performance on the questions participants actually encountered. My assessment is that this strengthens interpretation within the battery. It does not make that battery a representative sample of future work.
Two distinctions are essential when interpreting the reported gaps. First, the better component selected separately for every item is a demanding retrospective comparator. A person choosing an answer in real time would need some usable way to identify that component. Falling below this reference identifies an evaluation gap without establishing that users could have closed it with the information available.
Second, the pass-through comparison is a slope over a specified competence contrast. It should not become a claim that half of an assistant’s social value is wasted. Such a claim would require a definition of value, a feasible alternative workflow, and an account of costs. Accuracy responsiveness and recoverable benefit are different quantities.
The coding issue also reaches beyond statistical precision. Agreement with an assistant can arise through independent reasoning, persuasion, or copying. An adoption label cannot by itself distinguish those routes. For accountability, I would want records that separate an initial judgment, the advice received, the final judgment, and a reason for revision. That is a proposed evidentiary standard, not something an answer-agreement measure alone supplies.
Hypothetical example
Imagine a public-service worker checking an eligibility recommendation. The worker and assistant might agree because both correctly read a rule, or because the worker accepts the assistant’s mistaken reading. Identical agreement rates could accompany these different processes. An evaluation could introduce carefully controlled errors and test whether workers identify them, while separately checking whether workers have authority to pause the decision. This example extends the measurement question; it is not evidence about any deployed service.
The site’s Automation Bias article supplies broader oversight context. This review’s narrower concern is what observations would justify attributing a final answer to independent judgment. Accuracy evidence should inform that inquiry without substituting for an assessment of authority or responsibility.
Limitations
Original editorial analysis
The recruitment design limits causal interpretation. Neither an apparent assistant advantage nor a confidence difference should be translated into a claim that a particular model caused that outcome. A causal explanation would need a design capable of separating assistant identity from differences between participant groups and recruitment periods.
Similarly, a reference policy is an evaluation instrument before it is a practical intervention. A usable policy would need to identify reliability on unfamiliar questions without access to their answer keys. Repeated agreement among model outputs could be investigated as a signal, but agreement should not be assumed to establish correctness. The relevant question is whether the signal improves decisions when tested prospectively.
For governance, I would keep task accuracy separate from alignment with acceptable goals. A workflow could improve test performance while leaving unanswered who sets its objective, who may refuse its advice, and who bears responsibility for mistakes. Nor do these reasoning scores measure subjective experience or machine consciousness. They provide no evidentiary bridge to spiritual doctrines.
Proposed tests
A follow-up could randomly assign participants to answer-first, advice-first, and optional-consultation interfaces while holding the assistant and question distribution fixed. It could measure harmful revisions, successful corrections, confidence before and after advice, time, and perceived control. This would test whether requiring an initial judgment changes selective reliance, rather than assume that independent reasoning follows from an extra interface step.
A separate prospective evaluation could train a reliability signal on one question set and test it on new items. Its comparator should include unaided work, always following the assistant, and ordinary conversation. Independent validation of both advice correctness and adoption coding would help determine whether apparent improvements survive measurement uncertainty.
Finally, I would require a cost assessment alongside any proposed accuracy gain: human review time, model calls, latency, financial expense, and measured energy use where available. These are proposed reporting requirements, not resource findings from this study. They would allow a harder question about collective intelligence: whether a tested improvement warrants the labor and material infrastructure needed to obtain it.
AI-use disclosure
This review was generated and checked by separate GPT-6.1 Sol runs with medium reasoning. Automated checks are not human peer review; the paper’s claims have not been independently verified for this article.
Paper sources
Bibliographic metadata checked 2026-10-23.