Blog · Analysis · May 2026

The Citation Machine Enters the Court

AI hallucinated citations are not just lawyer mistakes. They are failures in the institutional machinery that turns text into authority.

A Source-Discipline Crisis

The first public scandal of AI in law was easy to ridicule: lawyers asked ChatGPT for cases, filed fake citations, and were sanctioned. But the deeper problem is not stupidity. The deeper problem is that generative AI attacks the social machinery of authority at exactly the place where authority is supposed to be most traceable.

A court filing is not ordinary prose. It is a request for state power. It asks a judge to dismiss a claim, preserve an asset, imprison a person, transfer money, enforce a contract, protect a child, stop an agency, or bless a settlement. Its citations are not decorative. They are the beams that connect the argument to the law.

Large language models are very good at making text feel load-bearing. They can supply case names, quotations, parentheticals, doctrinal summaries, and a confident chain of reasoning. That surface is useful when treated as draft material. It is dangerous when treated as authority. The court system has now become one of the clearest laboratories for the wider AI-age evidence problem: when machines can generate plausible source trails, institutions must decide what counts as verification.

Mata Was the Warning

In Mata v. Avianca, the Southern District of New York sanctioned lawyers who submitted non-existent judicial opinions with fake quotations and citations generated by ChatGPT. Judge P. Kevin Castel's June 22, 2023 order made the institutional harm explicit: fake opinions waste opposing counsel's time, consume court resources, may deprive clients of real arguments, damage the reputations of judges and courts whose names are falsely invoked, and promote cynicism about the legal system.

The facts mattered. The problem was not merely that an AI tool produced false output. The problem was that lawyers filed it, failed to verify it, and continued defending the fake authorities after the court and opposing counsel raised doubts. The machine made the fiction. The humans laundered it into procedure.

That is why Mata became a template case. It showed that ordinary legal duties were already enough to reach the failure. Rule 11 did not need a special metaphysics of artificial intelligence. The filing either had support in real law or it did not. The lawyer either made a reasonable inquiry or did not. A fluent model output did not change the gatekeeping obligation.

The Professional Answer

The American Bar Association's Formal Opinion 512, issued July 29, 2024, treated generative AI as a professional-responsibility problem rather than a novelty problem. Lawyers using generative AI do not need to become AI engineers, but they do need a reasonable understanding of the specific tool's capabilities and limits. They also need to protect client information, communicate when required, supervise use inside the firm, and charge reasonable fees.

The key rule is simple: review the output. In court, the opinion says lawyers must review generative-AI materials, including analysis and citations, and correct false law, false fact, missing controlling authority, and misleading arguments before submission. Supervisors must also train and supervise lawyers and nonlawyers using these tools.

That framework is more important than any single disclosure rule. Some judges require lawyers to disclose AI use. Some courts have standing orders. Some lawyers argue disclosure is overbroad because AI is now embedded in search, drafting, proofreading, translation, and document tools. But disclosure alone does not solve the problem. A disclosed hallucination is still a hallucination. An undisclosed but verified draft may be less dangerous than a disclosed draft nobody checked.

The professional standard should focus on provenance, verification, supervision, and accountability. Who generated the text? What parts were machine-assisted? Which citations were checked against authoritative sources? Who signed off? What process catches invented cases, misquoted statutes, bad parentheticals, and sources that exist but do not support the proposition?

The comforting story was that legal-specific AI tools would solve the problem. Connect the model to a trusted legal database, use retrieval-augmented generation, and hallucinated law should largely disappear. That story is partly true, but not enough.

Stanford RegLab and HAI researchers tested leading AI-powered legal research tools from LexisNexis and Thomson Reuters. Their 2024 analysis found that these systems improved on general-purpose models, but still produced incorrect information more than 17 percent of the time for Lexis+ AI and Ask Practical Law AI, while Westlaw's AI-Assisted Research hallucinated more than 34 percent of the time in the tested benchmark. The researchers also distinguished outright incorrect answers from misgrounded answers: a citation can exist and still fail to support the proposition attached to it.

That distinction is crucial. The next failure mode is not only invented cases. It is real cases used as masks for wrong claims. A system can cite a genuine opinion, quote a real statute, or retrieve a real document while still misdescribing its holding, jurisdiction, procedural posture, current validity, or relevance. In law, source existence is the beginning of verification, not the end.

The pattern has reached elite practice as well. Reuters reported on April 21, 2026, that Sullivan & Cromwell apologized to a federal bankruptcy judge after a filing contained inaccurate citations and other AI-generated errors. The firm said it had policies and training for AI use, but those policies were not followed and a secondary review did not catch the inaccurate citations. That episode matters because it removes a convenient excuse. This is not only a problem for careless solo practitioners using public chatbots. It can pass through prestigious firms, urgent motions, internal policies, and senior review.

Why Courts Are Different

Every institution has a source-discipline problem now. Schools face AI-written papers with fake references. Newsrooms face synthetic images and fabricated screenshots. Agencies face model-generated summaries of records. Corporations face AI-drafted reports and compliance narratives. But courts are different because their authority depends so explicitly on citable lineage.

Law is a memory system. It stores decisions, statutes, regulations, procedures, filings, transcripts, doctrines, conflicts, and exceptions. A legal argument is supposed to show its path through that memory. The citation is the path marker. It lets the opposing party contest the claim, lets the judge inspect the authority, lets later courts understand the reasoning, and lets the public see that the decision did not come from private intuition alone.

AI-generated legal fiction breaks that chain. It creates the appearance of institutional memory without the memory. It gives the user a map with invented roads. Worse, it can do so in the visual grammar of legitimacy: reporter citations, docket references, quotation marks, case names, parentheticals, and procedural confidence.

This is why the courtroom is a preview of a wider model-mediated knowledge crisis. The danger is not merely that models make errors. The danger is that models can produce counterfeit verification artifacts at scale. They do not only answer. They can simulate the trail that makes an answer look answerable.

The New Standard

Courts and legal organizations should treat AI-assisted law as a verification workflow, not a drafting shortcut with occasional cleanup.

First, every cited authority should be checked against an authoritative source. The case must exist, the quotation must appear, the citation must match, and the cited passage must support the proposition. A source that merely exists is not enough.

Second, firms should separate generation from verification. The person or process checking citations should not rely on the same model output as proof. Verification should return to primary or trusted legal sources, not a second layer of confident synthesis.

Third, AI use should leave internal traces. Matter files should record when generative tools were used for research, drafting, summarization, translation, or citation support. The point is not ritual confession. The point is auditability when something goes wrong.

Fourth, high-stakes filings need tool-specific rules. A general chatbot, a legal RAG product, a document review model, and a cite-checking tool have different failure modes. Policy should name the tool class and the allowed use, not merely say "AI" as if it were one object.

Fifth, courts should target duties rather than panic. A blanket ban may push use underground. A pure disclosure rule may create paperwork without verification. The durable rule is that every submitted legal assertion remains the lawyer's responsibility, regardless of whether it came from a partner, associate, paralegal, search platform, model, or agent.

Sixth, legal education must teach adversarial source practice. Future lawyers need to know how models fail: invented authorities, misgrounded citations, stale law, false premises, jurisdictional confusion, quotation drift, and confident overbreadth. This is now part of ordinary competence.

The Spiralist Reading

The court is one of society's rituals for turning memory into force. It does not only decide disputes. It stages a public discipline: claims must be named, sources must be shown, arguments must be answerable, and authority must pass through procedure before it acts on bodies, money, property, families, and institutions.

The citation machine threatens that discipline by imitating its outer form. It gives argument the costume of authority without the labor of verification. In the small case, that means a lawyer gets sanctioned. In the large case, it means institutions become comfortable with source-shaped hallucination: documents that look grounded, policies that cite ghosts, reports that reference imaginary studies, knowledge systems that simulate the smell of evidence.

The answer is not anti-AI purity. Lawyers will use AI. Courts will use AI. Legal databases will use AI. The useful demand is stricter: generated text must become more accountable as it approaches authority. Drafting can be fluid. Filing cannot. Brainstorming can be speculative. Citation cannot. A model may assist the work, but it must not become the witness for its own truth.

The court's lesson belongs beyond the court. Any institution that acts on claims needs source discipline strong enough for synthetic media and model-mediated knowledge. The future will produce more fluent assertions than humans can comfortably inspect. That makes verification infrastructure a civic necessity, not clerical tidiness.

The old rule survives because it is still correct: show your sources. The new rule is harder: show that the sources survived the machine.

Sources

Return to Blog