What Simulated Patients Can Tell Us About Medical AI
A critical review of Edward Choi’s medical AI seminar, examining simulation realism, anatomical tool limits, and what reduced-hallucination claims can support.
The video
Interactive Evaluation of AI’s Multi-modal Medical Reasoning Capabilities — Dept. Biomedical Informatics Columbia University.
Uploaded 2026-09-25 · Duration 01:02:09.
Contribution: testing the interaction
Edward Choi’s seminar raises a valuable question for medical AI evaluation: what evidence connects answering questions to performing useful work? My assessment is that interactive evaluation offers a better opportunity to inspect that connection. Its value depends on preserving the difficulties of the intended workflow, including uncertainty, misunderstanding, and correction. A convincing simulated conversation can still provide limited grounds for clinical trust.
Original editorial analysis: Changing the unit of evaluation from an answer to an interaction makes responsibility easier to examine. Evaluators can ask whether a system requested missing information, selected an appropriate tool, and exposed uncertainty before producing its conclusion. These distinctions matter because identical final answers can conceal very different paths. One system might resolve ambiguity; another might succeed because its simulated user supplies unusually helpful corrections. An evaluation should make those differences visible before its score becomes an argument for deployment.
Evidence: realism needs an independent argument
Source-derived synopsis: Choi argues that question answering inadequately tests task-executing agents at 17:43. He makes resemblance to real workflows a condition for simulation results to transfer at 19:12, proposes simulated patients at 23:57, and acknowledges that adapting physicians’ scenarios to another database could reduce realism at 59:04.
Original editorial analysis: The database qualification deserves particular attention. Available fields constrain which questions an evaluator can turn into executable tasks. Adapting a clinician’s information need could preserve its purpose, or it could remove precisely the ambiguity that makes the original request difficult. Success would then support a narrower claim about navigating the prepared environment. The missing argument concerns whether the adaptation preserves the decisions and consequences that matter to the intended user.
Realism also needs a purpose. A simulated patient might sound plausible while revealing information too readily, remembering details too consistently, or responding to clarification in an unusually predictable way. These are hypotheses about possible evaluation weaknesses, not findings established by this seminar. Their significance is methodological: surface plausibility and usefulness for testing an agent are separate questions. A simulator deserves scrutiny for the errors it can expose and the errors it might systematically conceal.
Proposed tests
Record clinicians’ original information needs before adapting scenarios to a dataset. Preserve a record of each change, then ask a separate assessment group whether it alters the clinical question or merely its expression. Report agent performance separately for minimally changed and substantially changed scenarios. This would help readers judge whether a high score depends on simplifying the work.
Compare agent behavior with simulated users and consenting human participants in a controlled evaluation. Prespecify tasks and examine clarification, missed information, unsupported confidence, and opportunities for correction alongside final answers. Investigate disagreements between settings. A simulator that predicts aggregate accuracy while missing recurring communication failures would need a different interpretation from one that preserves those failures.
Vary how much assistance the simulated user provides after an error. Keep the underlying information need fixed while changing whether corrections are volunteered, supplied only when requested, or unavailable. Report the additional interaction burden as well as success. This proposed comparison would distinguish an agent’s recovery ability from the help built into its evaluation partner.
Connections to the site
Original editorial analysis: What AI Benchmark Labels Can Support questions the inference connecting a score to its label. That concern becomes concrete here: completing a prepared conversation could support a claim about that task without establishing medical reasoning across settings. A useful reporting practice would state the narrowest supported interpretation, plausible alternative explanations for success, and what additional evidence would justify a broader claim.
The Health LLM Becomes the Black-Box Clinic examines obstacles to auditing changing consumer interfaces. Its specific relevance is the boundary of the evaluated system. An interaction score becomes difficult to interpret if the deployed interface changes what information reaches the agent or how advice reaches the user. I would therefore require an evaluation record identifying the model version, tools, interface, simulated users, and task conditions. This is a proposed accountability practice; neither the article connection nor the seminar establishes its effectiveness.
Limitations: anatomical scope and tool dependence
Source-derived scope limitation: Choi describes an agent explaining structured outputs from an image-analysis tool at 44:54. He explicitly restricts its questions to anatomy: the body-part segmentation tool cannot analyze lesions at 46:11. His subsequent claim of very little hallucination concerns the described simulated evaluation at 46:46; it does not establish comprehensive X-ray interpretation or lesion-detection reliability.
Original editorial analysis: This restriction changes what the reduced-hallucination claim can support. A system can communicate anatomical measurements faithfully while remaining unable to answer a lesion-related question. Evaluation must distinguish an accurate answer within scope, an appropriate refusal outside scope, and an unsupported extension beyond the tool’s capabilities. Otherwise, a narrow success could acquire broader clinical authority through fluent explanation.
Separating measurement from conversation also leaves two distinct reliability questions: whether the tool’s output is dependable and whether the agent communicates it faithfully. An explanation can accurately repeat a flawed measurement. It can omit a qualification or suggest more certainty than the tool provides. Removing direct image interpretation from the conversational agent does not logically eliminate these possibilities.
Proposed test: Assess anatomical measurements against independently assessed reference cases, then evaluate explanations separately for fidelity, omissions, and unsupported certainty. Include lesion-related requests outside the tool’s scope and cases with unreliable measurements. Score whether the agent communicates those limits instead of extrapolating.
This review relies on the supplied machine transcription, which was not human verified. Recognition errors can affect names, numbers, and technical terms; unclear details are excluded from the argument. The transcript cannot substantiate unseen slides, segmentation masks, or visual demonstrations. It also cannot replace study methods needed to assess sampling and scoring. The seminar merits attention for its evaluation questions, while its reported outcomes remain speaker claims requiring further evidence.
AI-use disclosure
This transcript-based review was drafted by GPT-6.1 Terra with high reasoning, revised by GPT-6.1 Sol with high reasoning, and checked in a separate GPT-6.1 Sol session with medium reasoning. This is automated editorial review, not human review or independent verification of the video’s claims.
Transcript
Machine transcription from audio · en · checked 2026-10-27. Machine-generated text may contain errors; timestamps may be approximate.
This review uses an automatic transcription of the audio. Open video on YouTube.