Blog · Review Essay · Modified August 12, 2026 · Last reviewed August 12, 2026

AI Snake Oil and the Belief Machine of Prediction

Arvind Narayanan and Sayash Kapoor's AI Snake Oil is best read as a book about evidentiary escalation. A demo establishes that an output can be produced. It does not establish that the output measures the intended real-world property, improves a decision, or deserves authority over a person.

For this review, AI snake oil is a product or policy claim whose evidence cannot support the capability, decision value, or institutional authority advertised for a named task, population, setting, and date. The belief machine is the surrounding pipeline—benchmark, marketing, procurement, workflow, and record—that upgrades a provisional result into something people are expected to obey.

The Book

AI Snake Oil: What Artificial Intelligence Can Do, What It Can't, and How to Tell the Difference was published by Princeton University Press on September 24, 2024. The publisher's hardcover record gives 360 pages and ISBN 9780691249131. In Princeton's December 2024 interview, Arvind Narayanan was identified as a computer science professor and director of the Center for Information Technology Policy, and Sayash Kapoor as a computer science graduate student. Those affiliations describe the publication context; the source list dates every later status claim.

The authors define the problem by advertising, not by essence: a system is snake oil when it does not work as represented and likely cannot meet that representation. This review keeps the accusation attached to a claim. Weak evidence for a video-based hiring inference does not show that every hiring tool is useless; strong evidence for image generation does not transfer to forecasting worker performance, illness, recidivism, or student success.

The book's taxonomy is its method. It separates predictive AI, generative AI, content moderation, recommender systems, existential-risk arguments, and the institutions that circulate AI myths. These categories use different targets, data, failure modes, and evidence. Collapsing them into one word lets success in one system become borrowed credibility for another.

The authors' later "AI as Normal Technology" essay extends the method: consequential technology can be analyzed through ordinary institutions, diffusion, incentives, and safeguards without treating it as destiny or a separate species. That frame does not minimize AI. It makes specific claims inspectable.

From Capability to Authority

AI Snake Oil is strongest when it refuses category inheritance. A fluent chatbot, a facial-matching system, a fraud classifier, a recommender, and a tool-using agent do not succeed for the same reason. A gain in one category supplies no evidence for another. The harder error, however, occurs inside a category, when evidence for one claim is silently promoted into a stronger one.

A useful claim ladder has five rungs:

  1. Artifact: the system can produce an output under stated conditions.
  2. Task performance: on a defined test, population, metric, and baseline, the output meets a measured level of performance.
  3. Construct validity: the score or label actually represents the real-world property named in the sales claim.
  4. Comparative decision value: a specified rule using the output improves outcomes over a feasible alternative after error costs and distributional effects are counted.
  5. Deployment legitimacy: the institution may responsibly give the system this role, with bounded authority, notice, recourse, monitoring, and a stop condition.

A demo can support the first rung. A well-designed benchmark may support the second. Neither by itself establishes the remaining three. This is why "human in the loop" is not a cure-all: if the target is invalid, the comparison is missing, or the reviewer lacks time and authority to disagree, the human merely ratifies an unsupported inference.

Before evaluating a claim, therefore, name the verb and the consequence. Is the system generating, detecting, ranking, forecasting, summarizing, recommending, or acting? What is the base rate? How was the target observed? What errors matter? What is the non-AI comparator? Who can contest the output, and what happens while a challenge is pending? The question is not whether machine learning appears somewhere in the stack. It is how far the available evidence permits authority to travel.

Prediction Is Not a Decision

The book's most important target is predictive AI in consequential settings. A prediction is an estimate about a defined outcome conditional on available data. It is not a future fact, a causal explanation, or a decision. The decision additionally requires a threshold, an action, values about competing errors, and responsibility for what follows. This is the distinction developed from another angle in Prediction Machines and the price of judgment.

Even the outcome may be less stable than the product language suggests. "Job success," "risk," "need," and "fraud" are constructed through measurement rules. Historical labels can encode earlier staffing, policing, treatment, lending, or reporting practices. Some outcomes are observed only after an institution acts: a detained person's behavior if released is unavailable by design. The primary research on human decisions and machine predictions treats this selective-observation problem as central, not as a detail that more data automatically erase.

Performance evidence can also be contaminated. Kapoor and Narayanan's peer-reviewed leakage review identified 22 prior papers spanning 17 fields that collectively documented 294 affected papers; their civil-war-prediction case study showed how correcting evaluation errors could erase an apparent advantage over older methods. That result does not prove complex models never help. It proves that a headline score is only as credible as the separation of training from evaluation, the comparator, and the path from data to claim.

Decision value demands a further test. Suppose a model predicts a target accurately. Which threshold triggers action? What is the cost of a false positive, a false negative, or delay? Does performance hold prospectively, under the distribution and workflow where it will be used? Does the intervention change the outcome later used as a label? Does the system improve results over a simpler rule, added staff, changed process, or no model at all? A single accuracy number cannot answer these questions; uncertainty must be connected to decision cost.

The political danger begins when an estimate becomes an operational fact. A score can rank a worker, trigger investigation, narrow care, or block an opportunity. The next database records the institution's action, and a later model may learn from that record as if it described the person rather than the policy. Error then becomes history. The affected person faces both the original inference and an institution that may no longer recognize it as contestable.

The Belief Machine

AI Snake Oil is also an account of why weak claims persist. The answer is not simple credulity. Each participant can receive a local benefit: a vendor sells certainty, a manager buys legitimacy, an investor acquires a growth story, a journalist gets novelty, and a policymaker gets urgency. No participant has to prove the complete chain from output to social outcome.

The belief machine has a repeatable sequence. A curated demo makes the artifact vivid. A benchmark compresses heterogeneous behavior into a portable number. Marketing promotes that number into a broad capability. Procurement converts the capability into an organizational commitment. Workflow converts the output into action. The resulting records make adoption look like validation. Once developers optimize to the test, the benchmark also becomes the curriculum, selecting which capabilities receive investment and which failures remain invisible.

Regulation and audit can unintentionally add another layer of reality merely by naming a product class. That does not make regulation a mistake. It means governance must regulate claims and effects without implying that a regulated label validates the underlying construct. A compliance file can show that a process occurred; it cannot, without deployment-matched evidence, show that the product works.

The authors' corrective is neither blanket rejection nor enchantment. Some systems work for specified tasks. Some are useful only with extensive human support. Some harms are present and measurable. The discipline is to keep those statements separate long enough for public judgment to survive the pressure to be early.

Current Context

The book appeared in 2024, but its test remains current because evaluation still lags deployment. In July 2026, Kapoor's Princeton final public oral, "The Missing Science of AI Evaluation," described a gap between laboratory performance and real-world utility, including evaluations of scientific applications and AI agents. An event abstract is not peer review; it is relevant here as the author's documented 2026 extension of the book's research program.

Enforcement supplies a narrower example. In August 2025, the FTC finalized an order against Workado requiring competent and reliable evidence before the company advertises the accuracy or efficacy of covered AI-detection products. The order resolves one company's substantiation case. It is not a general performance study of every detector, and it does not establish that regulation alone can validate a product.

The wider governance record must also be scoped. NIST says its AI Risk Management Framework 1.0 is voluntary and under revision. OMB Memorandum M-25-21 applies to covered U.S. federal agencies and requires controls for high-impact uses, not every public or private deployment. In the European Union, the official implementation timeline current on August 12, 2026 is phased: prohibitions and AI-literacy provisions began in 2025; transparency rules and enforcement for applicable provisions began on August 2, 2026; the current timeline places Annex III high-risk obligations in December 2027 and Annex I product obligations in August 2028. "The AI Act applies" is therefore not a sufficient statement of which duty applies to which system on which date.

Governance and Safety

The governance lesson is proportionality: authority must stop where evidence stops. A vendor's evidence package should function as a claim receipt, not a brochure. For a consequential use, it should record:

  1. Claim identity: system and model version, owner, date, intended use, prohibited uses, and the exact claim being evaluated.
  2. Target and population: how the outcome was defined and observed, relevant base rates, inclusion and exclusion rules, data provenance, and conditions under which the target ceases to be meaningful.
  3. Evaluation: prospective or properly held-out tests, leakage controls, feasible comparators, uncertainty, subgroup results where relevant, stress conditions, and known failure modes.
  4. Decision contract: threshold, permitted action, costs of competing errors, abstention rule, human authority, and the evidence needed before escalation.
  5. Operations and rights: version and change logs, deployment monitoring, notice, a usable challenge path, incident handling, rollback, and an expiry or revalidation date.

NIST's govern, map, measure, and manage functions supply a voluntary vocabulary for this lifecycle. OMB M-25-21 makes parts of it mandatory for covered federal high-impact uses, including pre-deployment testing, impact assessment, ongoing monitoring, operator training, appropriate human oversight, and review or appeal provisions in specified circumstances. The EU AI Act imposes different duties by system class and phase. These sources are not interchangeable, but all expose the weakness of a one-time benchmark detached from the deployed decision.

Human oversight must itself be tested. A reviewer needs relevant information, enough time, freedom to reverse the output, and a record of whether reversals occur. Otherwise the human is an accountability buffer, not a safeguard. Likewise, an appeal that arrives after a job, benefit, or liberty interest is irreversibly lost is not meaningful recourse.

Procurement should make substantiation inspectable. Contracts can require access to evaluation evidence, notice before material model or data changes, incident and override records, audit cooperation, exportable logs, and a disable or exit right. If the vendor cannot show that the target is measurable, the test matches deployment, and a harmed person can obtain timely review, the safe response is to narrow the use or refuse it—not to hide the gap behind a confidence score or certification badge.

The AI-Age Reading

The review shelf around this book traces opaque scoring, digital poorhouses, algorithmic bias, surveillance, labor platforms, dashboards, and metrics. AI Snake Oil contributes a portable test: what is the smallest claim actually demonstrated, and where does the institution add authority that the evidence did not earn?

That test matters for generative and agentic systems, but their claims should remain separate. Content quality, factual reliability, task completion, and safe delegated action are not synonyms. A model may write a persuasive answer yet cite a nonexistent source. It may complete a benchmark task yet fail under changed interfaces, longer horizons, adversarial inputs, or ambiguous stopping conditions. The 2026 evaluation work cited above reinforces the need to test agents under realistic conditions rather than inherit confidence from language fluency.

For delegated action, the claim receipt must add tool permissions, credential scope, data access, action logs, approval gates, spending or deletion limits, rollback, and an accountable owner. This is the gap between a model result and a real-world authorization explored in Escape from Model Land. If those boundaries cannot be named and observed, the product is selling delegation without control.

Governance artifacts can become snake oil too. A disclosure proves disclosure, not truth. A fairness metric proves a defined calculation, not justice. A model card proves documentation, not deployment fitness. An audit proves only what its scope, independence, evidence access, and consequences permit; otherwise the audit becomes a compliance interface. Safety requires a chain from evidence to a decision that can be blocked, changed, appealed, or stopped.

Where the Book Needs Care

The book's clarity is also its risk. "Works" and "does not work" can suggest cleaner boundaries than deployments provide. A system may be useful for drafting and unsafe for adjudication, reliable for one population and poorly calibrated for another, or accurate only because uncredited human labor catches its failures. The unit of analysis must remain the versioned claim in a specific workflow.

Readers should also resist turning "snake oil" into a universal insult. "This validation does not support inferring employee performance from facial movement" is stronger than "AI hiring is fake." "This model card does not establish clinical decision benefit" is stronger than "AI medicine is hype." Precision is the difference between critique and counter-hype.

The book disputes some dramatic future narratives, but that is not evidence that frontier-model security, misuse, concentration, labor effects, environmental costs, or increasingly delegated action can be ignored. The consistent standard is to identify a mechanism, state uncertainty, distinguish scenarios from observations, and update as evidence changes. Skepticism should not be a mirror image of promotion.

There is also a political limit to debunking. A product can fail as science and still succeed as management if it supplies a reason to discipline workers, deny benefits, intensify surveillance, or cut staff. Better literacy is necessary but insufficient. Enforceable rights, audit access, worker and community participation, public records, procurement leverage, and liability determine whether weak evidence loses power.

What This Changes

The recurring pattern is a causal chain. An institution selects a target because it is legible. A metric makes the target portable. A benchmark directs development toward the metric. Procurement attaches authority to the score. Workflow turns the score into action. The action becomes a record. A later system learns from that record as if it were an independent observation. This is how a model of the world can help manufacture the world it later claims to describe.

AI Snake Oil changes the appropriate response from fact-checking a slogan to auditing that chain. Separate model output, real-world inference, decision rule, and observed consequence. Preserve the distinction in an evidence layer that records versions, sources, uncertainty, human changes, appeals, and later outcomes. Test whether the metric remains aligned with the mission, and whether the institution is using the model to learn or to stop listening.

The book's deepest value is to lower the temperature without lowering the stakes. No claim about consciousness, divinity, or general intelligence is needed. Systems matter because institutions give their outputs roles in work, care, credit, education, security, and public administration. The remedy for snake oil is not generic trust or cynicism. It is evidence whose scope limits authority, plus rights and operational controls that make the limit real.

Source Discipline

This review uses Princeton University Press for edition metadata and Princeton sources for author context and the authors' stated framework. The Patterns article and NBER paper support specific methodological claims. FTC material documents allegations, a settlement, and a final order; it is not generalized into a technical verdict. NIST supplies voluntary risk guidance, OMB supplies federal-agency policy, and EU sources supply the regulation and its phased dates. The Amazon URL is a disclosed purchase link, not a bibliographic authority.

The July 2026 Princeton event description documents an active research direction; it is not treated as peer-reviewed proof. Internal links explain how this site's concepts connect, not as independent evidence for external facts. Dates are included because model versions, agency policies, legal phases, and author affiliations change.

No isolated output, benchmark, failed product, or enforcement action can establish the nature of AI as a whole. Each conclusion is scoped by system type, task, target, population, evidence date, deployment setting, and consequence. Analytical extensions in this review—especially the five-rung claim ladder and the belief-machine sequence—are the reviewer's synthesis rather than claims attributed to the book.

Sources

Book links are paid affiliate links. As an Amazon Associate I earn from qualifying purchases.

Buy on Amazon Browse Books

Return to Blog · Return to Books