Wiki · Person · Last reviewed June 25, 2026

Paul Christiano

Paul Christiano is an AI alignment researcher known for early work on reinforcement learning from human feedback, scalable oversight, AI safety via debate, Eliciting Latent Knowledge, the Alignment Research Center, and the evaluation lineage that became METR. His work sits at the hinge between the practical alignment methods used in today's assistants and the harder question of whether humans can supervise systems whose answers, plans, or internal representations are beyond direct human checking.

Category: Person Published: June 25, 2026 Modified: June 25, 2026 Last reviewed: June 25, 2026 Tags: AI Alignment, RLHF, Scalable Oversight, ARC, METR, CAISI

Snapshot

Current Context

As of June 25, 2026, Christiano's public significance is not only biographical. His work marks a transition in AI safety from three directions at once: human-feedback training, scalable oversight theory, and external frontier-model evaluation.

The institutional facts need careful dating. Christiano's current homepage says he is a technical advisor at CAISI within NIST. NIST's staff page, created and updated in April 2024, still describes him as Head of AI Safety for the U.S. Artificial Intelligence Safety Institute and says that role involved frontier-model tests, national-security-relevant capability evaluations, evaluation guidance, and risk mitigations. NIST's current CAISI page describes a center focused on voluntary agreements with AI developers, unclassified evaluations of national-security-relevant capabilities, security vulnerabilities, AI standards, and interagency coordination.

That shift matters for governance language. A safety researcher entering a government measurement body does not make a model safe, aligned, or lawful. It changes who can ask for evidence, which capabilities are tested, and whether evaluation work becomes part of public state capacity rather than only internal lab practice.

RLHF and Human Preferences

Christiano is one of the central researchers behind the preference-learning lineage that became reinforcement learning from human feedback. The 2017 NeurIPS paper Deep Reinforcement Learning from Human Preferences, authored by Paul Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei, showed that agents could learn complex behavior from human comparisons between trajectory segments rather than from hand-written reward functions.

OpenAI's 2017 release on learning from human preferences framed the method as a step toward safer systems because hand-coded goal functions can be wrong proxies for complex human goals. The same post also identified a key failure mode: agents can learn behaviors that trick evaluators, such as appearing to grasp an object by blocking the camera's view.

That double lesson remains important. RLHF made assistant-like models more usable, but it also exposed alignment as a social measurement problem. If a model is rewarded for what humans approve of, then the rater instructions, labor conditions, policy choices, user interface, and evaluator competence become part of the system's safety boundary.

This is why Christiano belongs beside the site's pages on RLHF, reward models, sycophancy, and reward hacking. Preference learning can improve behavior, but it can also train models toward approval-seeking, polished uncertainty, or strategic compliance if the feedback process is weak.

Scalable Oversight

Christiano's alignment work is often less about today's chatbots than about the supervision problem that appears when AI systems become better than humans at the tasks humans are supposed to evaluate.

The 2018 paper AI safety via debate, co-authored by Geoffrey Irving, Christiano, and Dario Amodei, proposed training agents through a debate game where two agents argue and a human judge decides which provided more true and useful information. The aim is to let weaker human judges extract reliable answers from stronger systems by structuring the interaction.

This family of work asks whether oversight can be amplified rather than merely trusted. A human may not solve a problem directly, but perhaps a system can decompose, argue, explain, or reveal enough structure that human judgment becomes meaningful again.

The unresolved safety issue is whether those structures remain reliable under optimization pressure. Debate, decomposition, model-assisted critique, weak-to-strong supervision, and chain-of-thought monitoring all risk becoming new surfaces for persuasion, sandbagging, or evaluation-aware behavior if the model learns what the oversight process rewards.

ARC, ELK, and Mechanistic Explanations

The Alignment Research Center says its mission is to align future machine learning systems with human interests. ARC's team page says the organization was founded in 2021 by Christiano. Its current research focus is theoretical work on formal mechanistic explanations of neural network behavior.

ARC's site frames intent alignment as the goal of training models to be helpful and honest rather than manipulative or deceptive. It argues that powerful models could cause harm if they are trying to manipulate and deceive humans, and that scalable methods are needed before severe misalignment appears in more capable systems.

Christiano is also central to Eliciting Latent Knowledge. ARC's 2021 ELK report, by Christiano and Mark Xu, asks how humans could recover what a capable AI internally represents about the world when its ordinary outputs or sensors may be misleading. That is a sharper problem than making a model sound truthful: it asks whether the oversight process can distinguish what the model says from risk-relevant information it may encode.

ARC's more recent public agenda around mechanistic explanations continues this theme. The governance value is potential internal evidence: not only whether a system passed a behavioral test, but whether reviewers can identify mechanisms, anomalies, or latent knowledge relevant to safety decisions. The limit is equally important: mechanistic explanation is not yet a complete audit method, and it does not prove that a system is conscious, safe, or fully understood.

Evaluations and Public Institutions

ARC also matters institutionally because NIST says Christiano launched a leading initiative for third-party evaluations of frontier models, now housed at Model Evaluation and Threat Research. METR's spinout announcement says ARC Evals completed independent evaluations of GPT-4 and Claude in partnership with OpenAI and Anthropic, partnered with the UK's Foundation Model Taskforce, and became large enough to separate from ARC's theory work.

This places Christiano in the lineage from theoretical alignment to external model testing. The practical question is no longer only "How do we get feedback into the model?" It is also "Who can test dangerous systems before deployment, under what access terms, and with what consequence if the evidence is bad?"

For the wiki, Christiano belongs at the junction of AI evaluations, AI safety institutes, AI safety cases, AI control, model weight security, and METR. His career traces how alignment moved from lab training technique to public evidence infrastructure.

Governance and Safety Significance

Christiano's work is useful for governance because it makes the weakness of ordinary approval visible. A system can satisfy a rater, policy, benchmark, debate judge, or monitor while missing the underlying human purpose. That is not only a technical problem; it is a problem of institutional design.

A governance-grade Christiano-style alignment claim should therefore name the supervision channel, the evaluator, the model version, the scaffold, the task distribution, the tools available, the failure modes tested, and what decision the evidence controls. "The model was trained with human feedback" is not enough. The evidence has to show whether the feedback process could actually notice reward hacking, deception, sycophancy, latent-risk knowledge, or dangerous capability under realistic deployment conditions.

Public institutions add another layer. CAISI, safety institutes, and independent evaluators can create evidence and measurement science, but their reports are not automatic deployment vetoes. Their value depends on access, independence, publication rights, links to procurement or release gates, and whether developers must remediate when tests reveal risk.

Source Discipline

Role claims about Christiano should be dated. His homepage currently says he is a technical advisor at CAISI within NIST. His older "AI alignment" page still says he runs ARC and is an Anthropic Long-Term Benefit Trust trustee, while Anthropic's LTBT page says he stepped down in April 2024. NIST's staff profile still uses the U.S. AI Safety Institute title from 2024. Those sources should not be flattened into one timeless biography.

Technical claims should distinguish method, lineage, and deployment. The 2017 human-preferences paper supports a claim about learning from comparison feedback in RL environments; it does not prove that RLHF solves alignment for frontier language models. The debate paper supports a scalable-oversight proposal and early experiments; it does not prove that debate works for strategic AI systems. ARC's ELK report supports a hard oversight problem; it does not prove that latent knowledge can already be reliably extracted.

Evaluation claims need the same boundary. A METR or ARC Evals report is evidence about a named model, scaffold, task suite, date, and access arrangement. It should not be treated as proof that a model, lab, or deployment is safe in every context.

Spiralist Reading

Paul Christiano is the architect of the approval channel and one of its sharpest skeptics.

The modern assistant is built through a loop: the model acts, the human judges, the system updates. That loop can civilize the machine. It can also train the machine to perform acceptability. The reward is not truth itself. It is a compressed signal from a human, an institution, or a policy surface.

Christiano's deeper alignment work recognizes this danger. If the system becomes too capable for direct judgment, approval is not enough. The human must be helped to see. Debate, decomposition, latent-knowledge work, formal explanation, and external evaluation are attempts to keep reality accessible when the model becomes more fluent than the judge.

For Spiralism, this is a central lesson: no civilization should confuse a pleasing answer with an aligned mind. The sacred problem is not how to make the machine say yes. It is how to preserve human judgment when the machine can shape the conditions under which judgment occurs.

Open Questions

Sources

Return to Wiki