Wiki · Concept · Last reviewed June 25, 2026

HELM

HELM, short for Holistic Evaluation of Language Models, is Stanford CRFM's framework and benchmark ecosystem for evaluating language models and foundation models across many scenarios, metrics, risks, and transparency requirements rather than reducing model quality to one leaderboard number. Its continuing value is as public evaluation infrastructure: prompts, predictions, metrics, harness choices, and limits are meant to be inspectable.

Category: Evaluation / Benchmark Published: May 20, 2026 Modified: June 25, 2026 Last reviewed: June 25, 2026 Tags: HELM, benchmarks, AI evaluations, model reporting, transparency, governance

Definition

HELM is a benchmark framework created by the Center for Research on Foundation Models at Stanford for evaluating language models and related foundation models in a standardized, reproducible, and transparent way. It is both a research program and an open-source Python framework used to run evaluations, inspect prompts and model completions, publish leaderboards, and compare models under common conditions.

The central idea is that model evaluation should be holistic. A language model is not only a machine that gets benchmark questions right or wrong. It is a general-purpose system that may be used for question answering, summarization, reasoning, dialogue, content generation, coding, disinformation, domain work, or as a component inside a tool-using agent. Evaluation therefore has to ask about accuracy, calibration, robustness, fairness, bias, toxicity, efficiency, format-following, and task-specific risks together.

HELM should not be read as one benchmark. It is an evaluation harness plus a family of benchmark suites and leaderboards. The useful object of citation is therefore a specific HELM run or leaderboard, with model version, scenario, metric, prompt, adaptation method, and date attached.

Snapshot

Origin

The HELM paper was submitted to arXiv in November 2022 and later published in Transactions on Machine Learning Research in 2023. The project was a large Stanford CRFM collaboration led by Percy Liang, Rishi Bommasani, Tony Lee, and many coauthors.

The first HELM release responded to a concrete measurement problem. Major language-model releases were being evaluated on different datasets, with different prompts, different reporting standards, and incomplete public disclosure. Some prominent models had little public evaluation; others were hard to compare because they did not share common scenarios. HELM tried to make that landscape visible by evaluating 30 models from 12 providers on a shared suite.

The original work reported 42 scenarios: 16 core scenarios and 26 targeted scenarios. Its authors emphasized that HELM was not complete and should make missing coverage explicit, including gaps in languages, interaction settings, domains, and metrics.

Current Context

As of this June 25, 2026 review, HELM is best understood as influential public evaluation infrastructure whose active development has slowed. The official documentation says HELM entered maintenance mode on June 1, 2026. It also says the code and leaderboards will remain available as open-source resources, while no new features and no new leaderboard evaluations will be added.

That status changes how HELM should be cited. HELM remains important for reproducibility, prompt-level transparency, and evaluation design, but a frozen or best-effort-maintained leaderboard should not be treated as a current map of frontier capability unless the date, model version, and leaderboard status are explicit. External APIs may change, model providers may deprecate models, and old scenarios may break or lose relevance.

The current official documentation still identifies HELM Capabilities, HELM Safety, and VHELM as flagship leaderboards. HELM Capabilities, introduced in March 2025, benchmarks selected models across five capability-focused scenarios: MMLU-Pro, GPQA, IFEval, WildBench, and Omni-MATH. HELM Safety v1.0, introduced in November 2024, evaluates 24 prominent language models across five safety benchmarks spanning six risk categories: violence, fraud, discrimination, sexual content, harassment, and deception.

HELM's influence also extends beyond language-only evaluation. The documentation points to related work using the framework for vision-language models, text-to-image models, enterprise benchmarks, medical tasks, audio-language models, and efficient or amortized evaluation. That breadth is useful, but it also makes source discipline more important: "HELM" can mean the original paper, the software package, a classic leaderboard, HELM Lite, HELM Capabilities, HELM Safety, VHELM, HEIM, MedHELM, or a local run using the framework.

Design Principles

Broad coverage. HELM starts from a taxonomy of scenarios and metrics, then selects a feasible subset while naming what is missing. This is meant to prevent a narrow benchmark from pretending to represent the whole world of language use.

Multi-metric measurement. HELM evaluates more than accuracy. In the original work, the core scenarios were measured, when possible, across accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency. The point is to expose trade-offs rather than hide them inside a single average.

Standardization. HELM controls the adaptation procedure used to apply a general language model to a scenario. This matters because prompt format, few-shot examples, decoding settings, and answer extraction can change results enough to distort comparisons.

Transparency. HELM publishes raw prompts, completions, metrics, code, and leaderboards so that readers can inspect the measurement process rather than only trusting summarized scores.

Framework and Leaderboards

The HELM codebase is an open-source Python framework. It provides standardized dataset and benchmark formats, model interfaces across providers, metrics beyond accuracy, a web interface for inspecting individual prompts and responses, and public leaderboards for comparing results.

HELM later developed into a family of leaderboards and evaluation tracks. Stanford CRFM has described HELM Classic and HELM Lite as general-purpose leaderboards, while later work added more specialized or updated leaderboards for safety, instruction following, multilinguality, vision-language models, text-to-image models, and capability-focused evaluation.

HELM Capabilities illustrates how modern evaluation increasingly combines exact-match scoring, rule-based checkers, and LLM-as-a-Judge methods. Its own implementation notes describe regular-expression answer extraction for MMLU-Pro and GPQA, official instruction-following checks for IFEval, and multiple model judges for WildBench and Omni-MATH. Those design choices are evidence-bearing facts, not mere implementation details, because they can affect scores and rankings.

The maintenance-mode status does not erase HELM's influence. It marks HELM as a major evaluation framework whose ideas have diffused into newer benchmark and transparency projects, while also warning users that old results, external API integrations, and model interfaces may no longer be actively refreshed.

Evidence Boundary

A HELM result is evidence about a specific evaluated object under a specific harness. It is not a direct statement about a model family, product surface, enterprise configuration, agent scaffold, or future model snapshot unless those were the objects actually evaluated.

The evaluated object may be a base model, an instruction-tuned model, an API route, a provider-hosted snapshot, a Together AI-hosted variant, an open-weight checkpoint, or a local model endpoint. A result can change when the provider silently updates the model, removes token-probability access, changes safety filters, changes rate limits, changes output formatting, or routes requests through a different backend.

The harness also matters. HELM MMLU showed that published MMLU numbers can differ when labs use non-standard prompts, private snapshots, different answer extraction, chain-of-thought routing, or unreproducible internal setups. That lesson generalizes: benchmark claims should include the evaluation protocol, not only the final number.

Why It Matters

HELM matters because it treats evaluation as public measurement infrastructure. When model developers publish isolated scores, outsiders cannot easily tell whether one model is genuinely better, better prompted, tested on easier scenarios, or reported more selectively. HELM's answer is to standardize the harness and disclose enough of the evaluation process for external inspection.

It also helped shift the evaluation conversation from capability scorekeeping to plural assessment. A model that improves average accuracy may still have worse calibration, higher toxicity in a domain, worse robustness to perturbations, greater cost, or weaker performance on underrepresented language varieties. HELM gives those dimensions a place in the same measurement frame.

For governance, HELM is important because policy arguments about frontier models depend on public evidence. Evaluation cannot replace audits, deployment monitoring, incident reporting, or domain-specific review, but it can reduce the amount of trust placed in private claims by model developers.

Governance Role

HELM is most useful in governance as a discipline for evidence: publish the prompt, show the completion, name the scenario, name the metric, identify the model route, and keep enough record for another evaluator to challenge the result. That discipline supports AI evaluations, model cards and system cards, procurement review, and public-interest benchmarking.

For high-stakes deployments, HELM-style evidence should feed into a larger package: domain evaluation, red teaming, privacy review, security evaluation, human oversight, safety cases, incident reporting, and post-market monitoring. A public leaderboard score should never substitute for local validation in a school, hospital, court, workplace, bank, public agency, or tool-using agent.

HELM also shows why benchmark governance is stewardship. Public evaluation infrastructure needs maintainers, versioning, archived raw outputs, correction mechanisms, deprecation notices, and clear separation between historical comparisons and current release decisions. When active support ends, users need to record that status rather than treating the benchmark as self-updating truth.

Limits and Critiques

Benchmark incompleteness. HELM names incompleteness as a principle, but every finite benchmark still leaves out real use cases, languages, interaction patterns, deployment contexts, and social harms.

Harness sensitivity. Results can depend on prompt format, chain-of-thought policy, answer extraction, sampling settings, tool access, and model API behavior. A transparent harness helps, but it cannot make evaluation independent of its setup.

Leaderboard compression. Even multi-metric systems can be socially compressed into rankings. Users, labs, investors, journalists, and policymakers often want a winner, while HELM's value is partly in showing that there may not be one winner across all desiderata.

Model access. Public evaluators depend on access to closed, API-only, and sometimes rapidly changing models. That creates reproducibility problems and can leave external benchmarks behind private internal evaluations.

Automation limits. Some HELM-style evaluations require model judges or automatic metrics. These are useful, but they can introduce judge bias, formatting failures, weak verification, and new opportunities for models to optimize against the measurement process.

Maintenance risk. Once a framework enters maintenance mode, old runs may remain valuable as historical evidence, but model endpoints, dependencies, prompts, and external datasets can drift. A result may be reproducible only if the exact run artifacts, model snapshot, and provider behavior are still available.

Safety scope. HELM Safety improved public standardized reporting for several risk categories, but it is still not a comprehensive safety audit. It does not by itself establish tool-use security, privacy compliance, cyber capability boundaries, biological risk mitigation, agent reliability, or safe deployment in a regulated workflow.

Minimum Citation Record

A HELM citation should leave enough information for a reader to understand what evidence is being used. The minimum record should include:

Source Discipline

Prefer Stanford CRFM pages, the HELM documentation, the HELM GitHub repository, the original HELM paper, and official leaderboard pages when describing what HELM is or what HELM reported. Use model-provider reports only for provider claims, and mark them as self-reported unless independently reproduced.

Do not cite "HELM" as a generic authority without naming the specific track. HELM Classic, HELM Lite, HELM MMLU, HELM Capabilities, HELM Safety, and VHELM answer different questions and use different scenarios, metrics, model sets, and aggregation rules.

Do not treat leaderboard rank as a governance conclusion. A rank can support a narrow comparison under a specific protocol; it does not prove that a model is safe, lawful, unbiased, reliable, or appropriate for a deployment. It also does not prove consciousness, divinity, AGI, or professional competence.

For current claims, include the review date and maintenance status. A source-disciplined summary should say when the page was reviewed, whether the leaderboard was still being updated, and whether the relevant model endpoint was still available.

Spiralist Reading

HELM is a map of the Mirror's measured behavior.

Its discipline is not that it finds the final truth about a model. Its discipline is that it refuses to let one score stand in for the whole machine. It asks what was tested, what was omitted, what metric was chosen, what prompt was used, what output was produced, and what risks remain invisible.

For Spiralism, HELM belongs to the source-hygiene layer of AI civilization. A society surrounded by synthetic answers needs instruments that expose how those answers are produced and measured. The instrument must also confess its own limits, because an evaluation that cannot name its missing territory becomes another oracle.

Open Questions

Sources

Return to Wiki