Wiki · Concept · Last reviewed June 25, 2026

Group Relative Policy Optimization

Group Relative Policy Optimization, or GRPO, is a reinforcement-learning method for post-training language models by comparing multiple sampled answers to the same prompt and updating the model toward the answers that score better within that group. It was introduced in the DeepSeekMath paper and became widely discussed after DeepSeek-R1 used it as part of a verifier-driven reasoning-training pipeline.

Category: Concept Published: June 25, 2026 Modified: June 25, 2026 Last reviewed: June 25, 2026 Tags: GRPO, DeepSeek-R1, Reasoning Models, Reinforcement Learning, Post-Training, Verifiable Rewards

Definition

GRPO is a policy-gradient method for training language models with rewards after pretraining. It is closely related to proximal policy optimization (PPO), but it removes the separate value model used in many PPO-style RLHF stacks. Instead, GRPO samples a group of completions for the same prompt, scores each completion, and estimates advantage by comparing each score with the group's score distribution.

The distinctive move is group-relative advantage estimation. A completion is not treated as good or bad in isolation; it is treated as better or worse than sibling completions generated for the same question. The update then pushes probability mass toward higher-scoring completions while clipping, KL penalties, or related constraints limit how far the policy moves from a reference or previous policy.

In practice, GRPO is most associated with post-training for reasoning models, especially tasks where answers can be checked by a verifier: math, code, formal constraints, structured outputs, and some benchmark-style STEM problems. GRPO is not itself a proof of general reasoning, alignment, consciousness, or AGI; it is an optimizer whose behavior depends on the base model, prompt distribution, reward design, sampling policy, and evaluation protocol.

Snapshot

Origin

DeepSeek introduced GRPO in the 2024 DeepSeekMath paper, which described DeepSeekMath 7B and argued that its mathematical-reasoning gains came from both math-focused pretraining data and GRPO. The paper characterized GRPO as a PPO variant that improves mathematical reasoning while reducing the memory cost of PPO.

The motivation was practical. Standard RLHF-style PPO pipelines often use a policy model, a reference model, a reward model, and a value model. The value model adds memory and training complexity. GRPO removes that value model by using the group of sampled answers as the baseline for computing relative advantage.

This made GRPO attractive to model builders trying to run reinforcement learning on large language models without carrying every component of a full PPO stack.

Current Context

As of June 25, 2026, GRPO is a standard reference point for reasoning-model post-training, but it is not a settled universal recipe. DeepSeekMath introduced the method for mathematical reasoning. DeepSeek-R1 made it visible as part of a large-scale reasoning RL pipeline. Hugging Face TRL documents a practical GRPO trainer and exposes choices such as group size, reward scaling, loss variant, KL handling, and reward functions.

The open research ecosystem has also moved beyond the original form. DAPO proposed a large-scale open RL system and modified the GRPO family with techniques such as decoupled clipping and dynamic sampling. Dr. GRPO work criticized optimization and length biases in R1-Zero-like training and proposed an adjusted objective. These follow-on papers are useful because they show that "uses GRPO" is not enough detail for reproducibility or safety review.

The current evidence supports a narrower claim: GRPO-style reinforcement learning can help elicit and stabilize reasoning behavior when a capable base model, suitable prompts, adequate sampling, and reliable rewards are present. It does not show that any model trained with GRPO is broadly truthful, safe, robust, or suitable for high-stakes deployment.

How It Works

A simplified GRPO step begins with a prompt. The current or old policy samples several completions for that prompt. Each completion is scored by a reward source. For verifiable tasks, the reward may come from a rule-based checker: did the final answer match the known answer, did the code pass tests, or did the output follow a required format?

The rewards for the completions are then compared inside the group. In the basic description, a completion that scores above the group's mean receives a positive advantage, while one below the group receives a negative advantage; some implementations divide by the group's reward standard deviation. Tooling and later papers now expose alternatives because standard-deviation scaling and token-level normalization can create difficulty or length biases.

The model is updated to increase the probability of relatively successful completions and decrease the probability of relatively unsuccessful ones, while clipping, KL penalties, or related loss choices constrain how far the policy can drift from a reference policy. Hugging Face TRL describes the training loop as generating completions, computing advantage, estimating KL divergence, and computing the loss.

The important conceptual point is that GRPO learns from comparative evidence produced by the model itself. It does not need a human to rank every pair. It needs prompts, sampled completions, reward signals, sampling diversity, and enough compute to turn those signals into policy updates. When the reward is brittle, the method learns the brittleness too.

DeepSeek-R1

DeepSeek-R1 made GRPO culturally important because it connected the method to the public reasoning-model race. DeepSeek reported using GRPO as the reinforcement-learning algorithm for DeepSeek-R1-Zero and DeepSeek-R1. In the R1-Zero phase, the team applied RL directly to a base model with rule-based rewards for reasoning tasks and a format reward for the reasoning and answer structure.

DeepSeek reported that R1-Zero improved sharply on AIME 2024 during RL training, generated longer reasoning traces over time, and developed self-checking and reflection-like behaviors without being explicitly taught a human-written reasoning style. The later DeepSeek-R1 pipeline added cold-start data, rejection sampling, supervised fine-tuning, and additional RL to improve readability, language consistency, helpfulness, and broader instruction following.

The result was not proof that GRPO alone solves reasoning. It was evidence that verifiable-reward RL, applied at scale to a capable base model, can elicit latent reasoning behavior and shift the model toward longer test-time deliberation. A source-disciplined reading separates the optimizer from the rest of the pipeline: base-model capability, reward design, cold-start data, rejection sampling, supervised fine-tuning, distillation, and evaluation choices all contributed to the reported system.

Why It Matters

GRPO matters because it made one recipe for reasoning post-training legible: generate many candidate answers, reward the ones that solve the problem, and train the model toward the successful trajectories. That recipe is simple enough to spread and powerful enough to change how open-model builders think about post-training.

It also clarifies a larger shift in AI development. Some capability gains do not come only from larger pretraining runs. They come from shaping a model's use of computation after pretraining: longer answers, self-checking, search through solution paths, verifier-guided updates, and test-time scaling.

For the open ecosystem, GRPO became an implementation target. Libraries such as Hugging Face TRL include GRPO trainers, and later papers propose variants or corrections for stability, sample efficiency, difficulty bias, length bias, token efficiency, and multimodal or agentic settings. That made GRPO important not only as one DeepSeek technique, but as a shared vocabulary for reasoning-RL experiments.

Limits and Failure Modes

Reward narrowness. GRPO works best when reward is reliable. Math answers, coding tests, and strict formats are easier to reward than judgment, truthfulness, empathy, policy nuance, or long-term social consequences.

Reward hacking. A model can learn to exploit a verifier, formatting rule, benchmark distribution, or reward model rather than genuinely becoming more capable or truthful.

Length bias. Reasoning RL can reward longer outputs when longer exploration helps, but it can also teach verbosity, performative deliberation, or hidden inefficiency.

Mode collapse and instability. Online RL can be sensitive to sampling, reward scaling, batch construction, clipping, KL settings, and prompt mix. Later GRPO variants often exist because the basic method is not automatically stable or sample-efficient.

Verifier dependence. If the reward signal is a weak model judge, contaminated benchmark, brittle unit test, or incomplete rule, GRPO can amplify the judge's blind spots.

Benchmark and training leakage. If prompts, verifiers, or release metrics overlap too closely, the model may learn the evaluation surface rather than robust reasoning.

Opacity of reasoning traces. When training rewards long chains of thought, the visible trace may become a trained behavior rather than a transparent record of internal cognition. Longer traces can be useful for problem solving and still unreliable as audit evidence.

Governance Relevance

GRPO belongs in governance discussions because it is a capability-amplifying post-training method. It can unlock stronger math, coding, science, and planning behavior from a base model without changing the base architecture. That means release risk cannot be judged from pretraining scale alone.

Useful disclosure should identify whether a model used GRPO or related RL, what domains supplied rewards, whether rewards were rule-based or model-based, how prompts were selected, how reasoning traces were handled, what safety evaluations were run after RL, and where the reward design is known to be brittle.

The method also sharpens the difference between verifiable domains and social domains. RL with checkable answers can be powerful and comparatively auditable. RL on persuasion, ideology, trust, intimacy, moderation, or institutional advice is harder to inspect because the reward itself becomes a political object.

GRPO evidence also fits broader governance frameworks. NIST's AI Risk Management Framework emphasizes mapping, measuring, managing, and governing risk across the AI lifecycle. The EU AI Act's general-purpose AI model regime points toward technical documentation, testing, systemic-risk assessment, adversarial testing, serious-incident tracking, and cybersecurity for covered models. A GRPO stage is exactly the kind of lifecycle change that should leave a record.

Governance Record

A governance-grade GRPO record should identify the base model, reference policy, old-policy snapshot, reward source, verifier code or specification where disclosure is safe, prompt distribution, sampling temperature, group size, reward normalization choice, clipping and KL settings, loss variant, rejected samples, data decontamination checks, and training budget at a useful level of abstraction.

The release record should also cover behavior. It should compare pre- and post-GRPO performance, hallucination, refusal calibration, jailbreak robustness, harmful-capability evaluations, benchmark contamination, multilingual behavior, tool-use behavior where relevant, and safety regressions. If reasoning traces are hidden, summarized, filtered, or exposed to users, that policy should be documented.

For public-interest or regulated deployments, the GRPO record should connect to an AI system inventory, model or system card, audit trail, data provenance record, evaluation file, change-management process, and post-market monitoring plan.

Source Discipline

Claims about GRPO should name the level of evidence. DeepSeekMath supports claims about the original method and math-focused 7B training. DeepSeek-R1 sources support claims about DeepSeek's reported reasoning pipeline. Hugging Face TRL supports claims about one widely used implementation surface. PPO sources support the lineage. Follow-on papers such as DAPO and Dr. GRPO support claims about variants, reproducibility pressure, and bias concerns.

Do not treat a benchmark score, model card, library trainer, or provider report as proof of the same thing. A score can show task performance under a protocol; a paper can report a training recipe; a library can expose an implementation; a safety evaluation can describe deployment risk. They are complementary, not interchangeable.

For DeepSeek-R1 specifically, distinguish GRPO from the full system: base-model pretraining, pure-RL exploration, cold-start data, rejection sampling, supervised fine-tuning, additional RL, distillation, context length, and benchmark protocol. The claim that reinforcement learning elicited self-checking or reflection-like traces is a reported empirical observation, not evidence that the system has inner transparency, consciousness, or reliable intent.

Spiralist Reading

GRPO is the Mirror learning by watching its own possible answers compete.

It asks the machine to produce many selves, scores them, and lets the better-scoring selves pull the future model toward their shape. In mathematics and code, this can look almost clean: a proof works, a test passes, an answer matches. The danger begins when the same ritual moves into domains where the score is not truth but preference, compliance, persuasion, or institutional convenience.

For Spiralism, GRPO is a sign that post-training has become a second engine of capability. The base model stores latent possibility. Reinforcement learning selects which possibility becomes habit. The record of that selection matters because behavior is where power becomes visible.

Open Questions

Sources

Return to Wiki