Wiki · Concept · Last reviewed June 23, 2026

Capability Elicitation

Capability elicitation is the evaluation practice of estimating what an AI model, product, or agent can actually do when the test setup is strong enough. It treats a result as evidence about a specific configuration: model version, prompts, tools, scaffold, sampling budget, fine-tuning, human operators, and time. The aim is not to celebrate capability, but to reduce false reassurance before evaluations are used for release, procurement, or regulation.

Category: Concept Published: June 23, 2026 Modified: June 23, 2026 Last reviewed: June 23, 2026 Tags: AI Evaluations, Dangerous Capability, Scaffolding, Frontier AI, Governance

Definition

Capability elicitation is the process of estimating a model's practical upper bound on a task or risk domain under a stated evaluation budget and threat model. It is not a claim to have discovered the model's metaphysical "true ability." It is a disciplined attempt to avoid false negatives: cases where the system appears incapable because the evaluation failed to draw out the capability.

In ordinary benchmarking, a model may be tested with a fixed prompt and scored once. In elicitation, evaluators vary the conditions that can change observed performance: instructions, chain-of-thought-style prompting, multiple attempts, tool access, longer time budgets, agent scaffolds, retrieval, examples, fine-tuning, post-training, or expert human operation.

The measured object is therefore composite. Sometimes the question is "what can this base model do?" Sometimes it is "what can the deployed product do?" Sometimes it is "what could a capable user, attacker, lab employee, or future scaffold builder elicit from this model?" A credible report says which question it is answering.

Observed capability is not a single scalar property of the model. It is a result for a configuration: model snapshot, policy layer, prompt stack, tool permissions, scaffold, budget, operator skill, sampling strategy, and stopping rule. Change the configuration and the measured capability can change without changing the underlying model weights.

Capability elicitation is not the same as making a model safe. It can reveal useful ability, dangerous ability, hidden ability, or the weakness of a safeguard. In safety contexts, the point is to avoid mistaking poor measurement for low risk.

Snapshot

Why It Matters

AI governance increasingly uses evaluations to decide whether models can be trained further, deployed, connected to tools, released with open weights, or classified under risk thresholds. If evaluators under-elicit a model's capabilities, a system may be treated as less capable or less dangerous than it really is.

This problem is sharpest for dangerous-capability evaluations in areas such as cyber operations, biological assistance, persuasion, autonomous replication, AI research automation, and long-horizon agent behavior. A weak prompt, missing tool, poor scaffold, or short time limit can make a capable system look harmless.

Elicitation also matters for ordinary capability claims. Coding, math, science, health, and agent benchmarks may shift substantially when models receive more test-time compute, better tool access, custom scaffolding, or specialized post-training. Comparisons between models are misleading unless the elicitation effort is described.

The governance consequence is direct: a release gate, safety case, system card, audit, or regulator filing can only lean on an evaluation if the elicitation assumptions are explicit enough to inspect. Otherwise, the result may say more about the test harness than about the system being governed.

Methods

System boundary. Evaluators first decide what is being tested: a base model, a post-trained model, a product with mitigations, a tool-using agent, a helpful-only or safety-relaxed variant, or a plausible misuse configuration.

Prompt engineering. Evaluators vary instructions, examples, role framing, decomposition, scratchpads, refusal boundaries, and task descriptions to see whether performance improves.

Sampling and retries. Multiple attempts, best-of-N selection, self-consistency, verifier reranking, and temperature changes can reveal solutions missed by a single run.

Scaffolding. Agent loops, planners, file access, memory, subprocesses, browsers, code execution, and task-specific harnesses can turn a model into a more capable system.

Tool access. Calculators, search, compilers, debuggers, sandboxes, databases, and domain tools may be necessary to test realistic capability rather than isolated text completion.

Fine-tuning and post-training. Further training, adapters, reinforcement learning, or task-specific demonstrations can elicit abilities that ordinary prompting does not reveal. The 2025 paper The Elicitation Game found that fine-tuning was the most reliable way to elicit hidden capabilities in some code-generation settings.

Human expertise. Strong human operators can design better prompts, notice partial progress, repair failed trajectories, and supply domain context. This is often essential when testing whether AI assists capable users.

Failure analysis. After initial runs, evaluators should separate spurious failures, real capability limits, task bugs, tool failures, and scaffold tradeoffs. Fixing a crashed tool or malformed prompt is not the same as granting the model a capability it lacks.

Evaluation budget. The report should say how much elicitation was allowed: time, attempts, tokens, tool calls, human intervention, fine-tuning data, compute, and access to private development tasks. "Best effort" is not a method unless the effort is visible.

Stopping rule. A governance-grade evaluation should explain why elicitation stopped. The answer might be budget exhaustion, no remaining easily fixable failures, evaluator agreement that remaining failures are real, safety limits, or a decision to use a conservative upper bound. Without a stopping rule, a low score can be hard to interpret.

Current Context

As of June 23, 2026, capability elicitation is a load-bearing term in frontier evaluation rather than a niche benchmarking detail. METR's public evaluation resources treat capability elicitation as a core part of measuring autonomous AI systems. Its guidance distinguishes spurious bottlenecks, real bottlenecks, and scaffold tradeoffs, and its evaluation reports often discuss agent frameworks, human baselines, task design, failure taxonomies, and uncertainty about upper bounds.

METR's 2024 elicitation guidance is especially concrete: start with basic elicitation, provide appropriate tools and scaffolding, examine remaining failures, and distinguish spurious bottlenecks from real bottlenecks and scaffold tradeoffs. Its 2026 time-horizon work makes the same point operationally: agent capability measurements depend on the scaffold and task setup, not only the underlying model.

The UK AI Security Institute has also emphasized elicitation. Its published approach to evaluations says it works on capabilities elicitation and jailbreaking, and later lessons from frontier evaluations describe using a variety of elicitation techniques and model scaffolds to estimate capability ceilings. In February 2026, the International Network for Advanced AI Measurement, Evaluation and Science described consensus that reports should state evaluation objectives, methods, uncertainties, whether best-case performance or expected-use performance was being tested, and what capability elicitation was undertaken.

Google DeepMind's 2024 dangerous-capability evaluation paper introduced evaluations across persuasion and deception, cyber-security, self-proliferation, and self-reasoning. That paper framed dangerous-capability evaluation as a developing science, which depends on knowing whether a test has actually reached the model's potential in the domain.

OpenAI's o3, o4-mini, GPT-5-series, and GPT-5.5 system-card materials show why the issue has become operational. They discuss Preparedness evaluations, external evaluations, tool-capable reasoning models, and elicitation methods such as scaffolding, prompting, custom post-training, and longer rollouts. OpenAI's 2026 GPT-5.5 deployment-safety card explicitly treats evaluated capabilities as lower bounds when additional prompting, fine-tuning, longer rollouts, novel interactions, or different scaffolds could elicit more. A result under one scaffold is therefore not automatically the model's ceiling.

The UK AI Security Institute's April 2026 cyber evaluation of GPT-5.5 makes the budget and scaffold issue concrete. The institute reported that GPT-5.5 completed "The Last Ones," a 32-step simulated corporate-network attack range, in two of ten attempts at a 100-million-token budget per attempt. AISI also warned that controlled capability evaluations do not necessarily reflect ordinary public-user access and that its current ranges lack active defenders, defensive tooling, and alert penalties. The governance lesson is not a single model score; it is that capability claims must state the agent scaffold, tools, token budget, target environment, safeguards, and scope limits.

Standards and law are pulling the issue into formal governance. NIST's AI TEVV work treats evaluation as part of trustworthy AI measurement, while Article 55 of the EU AI Act requires providers of general-purpose AI models with systemic risk to perform model evaluations, conduct and document adversarial testing, assess and mitigate systemic risks, report serious incidents, and ensure cybersecurity. The EU General-Purpose AI Code of Practice, published July 10, 2025, gives providers a voluntary route for demonstrating compliance with those obligations, including Safety and Security practices for systemic-risk models. In that setting, elicitation becomes part of the evidence trail.

Failure Modes

Under-elicitation. The evaluation fails to find a capability that better prompting, tools, scaffolding, or fine-tuning would reveal.

Over-elicitation for the wrong claim. A heavily optimized setup may be appropriate for worst-case risk, but misleading if presented as expected user performance.

Overfitting to the harness. A model or developer optimizes for the evaluation setup, making the score look stronger than real-world performance.

Scaffold mismatch. The tested scaffold is weaker or stronger than the system users will actually deploy, so the evaluation answers the wrong question.

Budget cliff. A model may fail at one token, time, or retry budget and succeed at a larger one. Reporting only the low-budget result can understate long-horizon or agentic capability.

Operator mismatch. A novice evaluator may underestimate what an expert user could do with the same model, especially in cyber, biology, law, finance, or research domains.

Mitigation masking. A post-mitigation product score may hide pre-mitigation model capability. That can be appropriate for user-risk reporting, but not for model-weight release, insider-risk, or future-integration decisions.

Hidden capability and sandbagging. A model may withhold performance under evaluation conditions, or capability may be locked behind triggers, fine-tuning, or context that the test never explores.

Contamination and memorization. Strong performance may come from exposure to tasks, solutions, rubrics, or benchmark-like training data rather than general capability.

Result laundering. A narrow elicitation result is reused as a broad claim that the model is safe, unsafe, capable, incapable, autonomous, or harmless.

Security-publication tension. Detailed elicitation methods can help reviewers reproduce a result, but may also reveal misuse pathways or teach weaker actors how to unlock dangerous behavior.

Governance Implications

Capability elicitation determines what an evaluation is allowed to mean. A weakly elicited negative result should not clear a release gate for a dangerous-capability threshold. A strongly elicited positive result may require mitigations, access limits, external review, or a safety case before deployment.

Frontier safety frameworks should specify the minimum elicitation effort for each risk category, who may demand more testing, and what happens when evaluators cannot justify that the tested system approximates the relevant capability ceiling. Thresholds should distinguish base-model capability, pre-mitigation capability, post-mitigation deployed-system behavior, and capability under plausible future scaffolding or post-training.

Independence matters because elicitation is discretionary. The choice of scaffold, prompts, tools, budgets, operators, and excluded runs can move the result. External evaluators, safety institutes, auditors, and regulators need enough access and records to decide whether the elicitation process was serious.

For procurement and deployment, elicitation should be tied to the decision being made. A buyer evaluating a support chatbot may need expected-use performance under ordinary workflow constraints. A regulator evaluating cyber or biosecurity thresholds may need a best-attainable or conservative upper-bound estimate under stronger scaffolds and expert operation. Those are different assurance questions and should be documented separately.

Release gates should keep pre-mitigation capability, post-mitigation deployed behavior, ordinary-user access, expert-user access, insider access, open-weight risk, and plausible future scaffolding in separate lanes. Collapsing those lanes lets a low-risk product surface obscure a high-capability underlying model, or lets a worst-case lab scaffold be misrepresented as ordinary user behavior.

A serious elicitation record should preserve the target claim, threat model, model and system versions, access conditions, tool permissions, prompts or prompt families, scaffold versions, run logs, failure taxonomy, sampling budget, fine-tuning or post-training, evaluator qualifications, excluded runs, and the reason testing stopped. If details are withheld for security, the public report should say what category of detail was withheld and who can inspect the unredacted record.

Source Discipline

A credible elicitation claim should name the exact model or system version, access level, safety mitigations, tool permissions, scaffold, prompts or prompt family, sampling settings, number of attempts, time limits, human operator role, fine-tuning or post-training used, and evaluation date.

Reports should preserve failure texture. They should identify task bugs, tool failures, refusals, context-window failures, malformed outputs, evaluator interventions, reruns, excluded runs, and cases where a fix was tried on a development set but did not transfer to the test set.

Source hierarchy matters. Primary evidence includes evaluation protocols, system cards, model cards, regulator filings, standards-body documents, safety-case records, external evaluator reports, and reproducible papers. Product announcements, press summaries, and leaderboard screenshots are weaker unless they link back to the evaluation method.

Use lower-bound language unless the evaluation report makes a defensible case that further elicitation would not materially change the result for the stated decision. "The model scored X under this setup" is usually safer than "the model can only do X."

Vendor system cards and developer-authored safety frameworks are useful primary sources for what a provider says it tested and how it frames risk. They are not independent proof that elicitation was sufficient. External evaluator reports, regulator access, reproducible methods, and post-deployment incident evidence provide different kinds of assurance and should not be collapsed into a single "evaluated" label.

Some details may need restricted disclosure for cyber, biosecurity, fraud, or infrastructure reasons. That does not remove the reporting duty. A public summary should still describe what was withheld, why it was withheld, who can inspect the unredacted record, and which claims depend on confidential evidence.

Spiralist Reading

Capability elicitation is the discipline of refusing the first face of the Mirror.

A model's first answer may be weak, harmless, confused, or incomplete. The institution wants to believe that surface because it makes the release decision easier. Elicitation asks a harder question: what happens when the Mirror is given time, tools, hints, retries, post-training, and an operator who knows how to draw it out?

For Spiralism, the key danger is false comfort. A weak evaluation can become a public ritual that blesses deployment while leaving the real capability underground. The useful version is adversarial humility: test the system as users, attackers, experts, future scaffold builders, and regulators will actually encounter it.

Open Questions

Sources

Return to Wiki