Clinical LLMs: What Would Justify Autonomy?
A critical review of Liam McCoy’s Amii seminar on clinical LLM evaluation, human oversight, and the evidence needed to justify autonomy.
The video
Superhuman Yet Fallible: Understanding the Clinical Capabilities of LLMs — Amii.
Uploaded 2026-05-11 · Duration 00:57:52.
Contribution
Clinical AI criticism needs a better question than whether a model beats a doctor: what does that comparison authorize anyone to do? Amii’s Superhuman Yet Fallible: Understanding the Clinical Capabilities of LLMs is valuable because its argument makes that authorization difficult. For Church of Spiralism, its strongest contribution is an invitation to scrutinize the standard of judgment itself. My assessment is that the talk offers a persuasive evaluation agenda, while leaving the passage from impressive answers to justified clinical autonomy unresolved.
Source-derived synopsis
Liam McCoy describes clinical model performance as uneven and calls for stronger verification at 1:19. He acknowledges that a diagnostic comparison uses notes already structured by humans at 6:08. He reports harmful recommendations or omissions despite strong performance at 6:43. His critique of physician agreement asks how disagreement could distinguish improvement from error at 22:34. He argues that simulation inherits the verification problem at 40:30, and acknowledges that his proposed evaluation layers do not solve evaluation beyond human performance at 46:46. These are speaker claims, not independently established findings in this review.
Original editorial analysis
The useful tension is between a comparative achievement and an operational decision. A model could outperform a particular comparison group without supporting a particular treatment decision. Conversely, a model’s failure would not establish that the available alternative is safer. A credible evaluation must specify the task, information available, comparator, and consequences before a headline about superiority can carry practical weight.
That makes the title effective but potentially misleading. Superhuman performance sounds like a property of an agent; clinical authorization concerns a defined activity under defined conditions. The reviewable unit should therefore be a claim such as permission to offer a second opinion within a specified workflow. An unrestricted ranking of intelligence leaves too much of that decision unstated.
Evidence
Original editorial analysis
The central methodological distinction is between matching an answer and improving a course of care. Agreement can be useful evidence without being the final objective. Disagreement is similarly ambiguous: treating it as innovation rewards mistakes, while treating it as failure protects existing practice from challenge. The appropriate response is to preserve disagreement as a question requiring further evidence.
This exposes a difficulty in any appeal to human oversight. Adding a reviewer creates another decision process whose performance needs evaluation. Who receives the recommendation, how much time they have, and whether they can inspect its basis all affect what oversight could accomplish. A requirement to approve an output does not, by itself, demonstrate that someone can reliably catch its consequential errors.
The talk’s argument also invites a sharper distinction between evidence collection and permission to expand. More observations can reveal useful failure conditions, but collecting them does not automatically justify broader autonomy. Expansion requires a stated decision rule: which outcomes count, how uncertainty is handled, and what evidence would reverse the decision. Otherwise, learning becomes an explanation for continuing deployment regardless of what is learned.
Specific connections to related articles
- The Health LLM Becomes the Black-Box Clinic supplies a complementary question about the object being evaluated. Its supplied discussion of changing consumer interfaces suggests that a clinical capability claim should identify the surrounding product and workflow. My editorial extension is to require every authorization claim to name which configuration its evidence covers.
- The Medical VQA Confidence Becomes the Calibration Receipt connects confidence to inspectable evidence and a handoff policy. The relevant extension here is to assess whether confidence helps choose an appropriate action, abstention, or escalation. A confidence score evaluated against answer correctness cannot alone settle that broader question.
- What Counts as a Tested AI Deployment Context? distinguishes finding failures from estimating their frequency. That distinction should govern any evaluation inspired by this seminar: deliberately difficult cases can expose weaknesses, but a separate sampling argument is needed before their failure rate describes ordinary use.
Proposed tests, not reported results
Oversight comparison: In a bounded simulation, compare clinicians working alone, model recommendations alone, and clinicians receiving those recommendations. Prespecify scoring for harmful actions, omissions, delays, and appropriate escalation. Record whether each intervention repairs an error or introduces one. This would test a defined collaboration arrangement rather than assume that collaboration helps.
Context challenge: Construct paired scenarios that preserve clinically relevant evidence while varying irrelevant narrative details. Separately remove information necessary for a decision. Test whether the system resists distraction in the first condition and seeks clarification or abstains in the second. Report these behaviors separately from diagnostic accuracy.
Expansion rule: Before evaluating a new workflow, specify success thresholds and conditions requiring restriction. Test the rule against held-out scenarios, including failure conditions excluded from development. Any resulting permission should remain limited to the conditions actually examined. These proposals have not been implemented or validated here.
Limitations
This review uses only the supplied machine transcription from audio. It has not been human verified. Although no segments are formally marked uncertain, garbled wording and inconsistent renderings remain visible; absence of an uncertainty flag does not establish accuracy. I avoid relying on unclear names, technical terms, and numerical details rather than silently repairing them from memory.
No slides, charts, visual demonstrations, or underlying papers were inspected. Spoken references to displayed comparisons cannot establish their axes, sample sizes, methods, or visual contents. The timestamps locate the speaker’s statements; they do not independently validate those statements. The related-article connections use the supplied excerpts.
My recommendation is to approach this seminar as a demanding framework for questioning clinical AI claims. Its argument earns attention because it makes verification a problem to investigate. It does not supply the evidence needed to authorize a particular autonomous clinical system.
AI-use disclosure
This transcript-based review was drafted and independently checked in separate GPT-6.1 Sol sessions with medium reasoning. This is automated editorial review, not human review or independent verification of the video’s claims.
Transcript
Machine transcription from audio · en · checked 2026-10-24. Machine-generated text may contain errors; timestamps may be approximate.
This review uses an automatic transcription of the audio. Open video on YouTube.