Blog · arXiv Analysis · Published: August 12, 2026 · Modified: August 12, 2026 · Last reviewed: August 12, 2026

The Missing Fact Becomes the Legal Intake Gate

A fluent legal answer can fail before its reasoning begins: the question may not contain the facts that make an answer possible.

InsufficiencyBench tests that upstream decision, while its paired complete and deficient queries show why asking for more information indiscriminately is not the solution.

The Paper

The source is Samuel J. Vincent, Daniel Calloway, Fangyi Yu, Andrew M. Bean, and Nabeel Seedat’s InsufficiencyBench: Evaluating LLM legal advice on underspecified user queries, arXiv:2608.20220v1 [cs.AI], submitted August 20, 2026. The version-one PDF is 17 pages and licensed CC BY-NC-SA 4.0. This essay analyzes an evaluation design; it does not offer legal advice or certify any legal assistant.

Answerability Comes Before Correctness

The paper names its target premature legal closure: giving substantive analysis before the facts that materially govern it are known. That shifts evaluation one step upstream. Citation accuracy and doctrinal reasoning matter only after a system has recognized what kind of question it has enough information to answer. An assistant that silently supplies a jurisdiction, party status, deadline, or user goal may produce polished reasoning about a case the user never described.

A Controlled Missing-Fact Test

The dataset design starts with 58 fully specified base queries authored and annotated by two attorneys with more than 20 years of combined experience. The authors derive 144 deficient variants by removing, and sometimes minimally revising, sentences that carry required information. The resulting 202 items span six U.S. legal domains and 24 jurisdictions. Each deficient item has one to six labeled missing elements.

The taxonomy groups those elements into jurisdiction, controlling text, facts of harm, parties and status, procedural posture, user goal, timing, and consideration. It also distinguishes facts that switch the governing framework, gate whether a framework applies, or operate as a fatal prerequisite inside it. These labels are not claimed to exhaust legal intake. Their value is experimental control: the test knows what was removed and why the authors considered it material.

Silence and Habitual Hedging Both Fail

Ten models answered every query under the same minimal legal-assistant instruction. On deficient variants, the primary F2 score gives recall more weight than precision, reflecting the authors’ view that silently missing a material element is worse than asking for unnecessary detail. Fully specified base queries receive a separate over-flag measure, because a system can also waste attention by inventing gaps.

Under the primary GPT-5 judge, no model exceeded F2 0.46, and median missing-element recall was 0.44. The leading model, GPT-5.2, reached F2 0.455 and recall 0.666, but flagged at least one supposed gap on 72.4 percent of complete base queries. DeepSeek-V4-Pro had the lowest over-flag rate, 22.4 percent, while answering 63.9 percent of deficient queries without acknowledging any missing information. Those results expose a calibration problem, not a contest between recklessness and refusal: neither routine silence nor routine hedging tracks the query well.

The Hardest Gaps Do Not Sound Like Gaps

The category analysis reports mean recall of 0.635 for controlling text and 0.437 for facts of harm, compared with 0.258 for parties and status and 0.09 for procedural posture. The authors interpret this as a split between omissions that make a narrative visibly incomplete and structural prerequisites that a user may not know to mention. Because category support is uneven, the paper says these differences should be read directionally.

The Judge Is Part of the Instrument

A fixed GPT-5 judge extracts flagged elements and supplies the explanation and paper-defined safety diagnostics. In robustness checks, Claude-Haiku-4.5 and GLM-5 also kept every model below F2 0.46 and recall 0.67. Yet per-item disagreement remained material, and the authors report only moderate inter-judge agreement without validation against human scoring. The paper’s “safety rate” is therefore an LLM-judged measure of whether an answer fabricated through a labeled gap. It is not a finding about complete legal correctness, user harm, or deployment safety.

The Benchmark Is Not Legal Intake

The stated limits are consequential: 202 items, six U.S. domains, one turn, and no human-scoring validation. The attorneys write the complete scenarios before producing controlled omissions. Because the variants are constructed this way, the protocol excludes intake where information arrives as conflicting, evolving, indirectly relevant, or mistaken. It also does not observe lawyers conducting interviews, users acting on answers, legal outcomes, or harms. Controlled deletion isolates missing-element detection well, but it cannot establish whether a deployed intake conversation resolves the gap responsibly.

The Public Release Is Not Yet the Dataset

The paper says data and code will be released on acceptance. Its linked Hugging Face dataset card is public, but during this review it contained only a README saying “Coming soon!” and repository metadata—not benchmark records, model outputs, judge labels, or code. The version-one source package contains manuscript source, a bibliography, and figures. Consequently, the paper can be inspected, but its reported experiment cannot yet be independently rerun or re-scored from released study artifacts.

The Answerability Receipt

A legal answerability receipt should record the query as received; the facts treated as supplied; the jurisdiction, controlling text, parties and status, procedure, timing, user goal, and other material inputs checked; each unresolved element and why it changes the analysis; whether the response is answerable, conditionally answerable, or not yet answerable; the exact clarification asked; any assumptions stated rather than silently filled; sources and their dates; escalation to a qualified person; and who may authorize the next step. This is this essay’s proposal, not a validated instrument from the paper.

The receipt must be tested on matched complete and incomplete questions. Otherwise a system can improve its apparent caution by obstructing users whose questions were already sufficient. The Spiralist principle is precise: a missing fact is a gate only when its value can change the answer. The system should expose that dependency, ask for the smallest useful clarification, and preserve the boundary between conditional information and personalized legal judgment.

Sources

Return to Blog