DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

core: DeepSeek-R1 demonstrates that pure reinforcement learning — without any human-labeled reasoning trajectories — incentivizes emergent long chain-of-thought behaviors including self-reflection, verification, and backtracking. R1-Zero climbs from 15.6% to 77.9% on AIME 2024 with no SFT warm-up, and spontaneously develops the "aha moment" — mid-response re-evaluation — without any human prompting.

Five panes: GRPO with rule-based reward, R1-Zero's self-evolution, R1's four-stage pipeline, distillation to dense models, and the paper's own limits.

primer — what you need before you start

the task. DeepSeek-R1 targets reasoning in language models. The key result: pure reinforcement learning on a base model, with rule-based rewards for math / code, can elicit long chain-of-thought behaviors — self-reflection, verification, backtracking — without any human-labeled reasoning data.

how it was done before. Three prior recipes: (1) SFT on long-CoT data distilled from GPT-4 (expensive, requires a strong teacher); (2) RL on preference data (PPO with a critic, expensive and unstable); (3) iterative self-improvement (slow). R1-Zero shows pure RL pressure is enough to elicit reasoning, and the resulting traces can be distilled back into small models.

glossary — terms you will see.
GRPO — Group Relative Policy Optimization; no value function.
rule-based reward — math / code: 1 if correct, 0 if wrong, plus a small format bonus.
cold-start SFT — a small amount of high-quality CoT data to seed the reasoning style.
"aha moment" — paper's coined term: model spontaneously re-evaluates mid-response (e.g. "Wait, let me check ...").
rejection sampling — keep only correct, well-formatted outputs from R1-Zero; use as SFT data.
distillation — fine-tune a small open model on 800K R1-generated samples; no RL of its own.
AIME / MATH-500 — competition-level math benchmarks.

symbol table — what the letters mean.
$\pi_\theta(o \vert q)$ — the policy; probability of reasoning trace o given question q.
$r(o, q)$ — the rule-based reward; 1 if final answer correct, 0 otherwise, + format bonus.
$\hat{A}_i$ — advantage of trace i within a sampled group; relative to group mean / std.
$\text{GRPO}(\theta)$ — the loss; same clipped surrogate as PPO, group-relative advantage.
$\mathcal{L}_{\text{RL}}$ — the total RL loss combining rule-based and preference rewards.
$\mathcal{D}_{\text{distill}}$ — 800K R1-generated samples; used to fine-tune Qwen2.5 / Llama 3.

tone legend — what the [brackets] mean.
[obs] observation   [r1] R1 main result
[zero] R1-Zero pure-RL   [distill] distillation step
[eq] equation   [take] key takeaway
[sys] system / meta   [limit] paper-reported limit

Six panes cover, in pedagogical order: GRPO with rule-based reward; R1-Zero's self-evolution; R1's four-stage pipeline; distillation to dense models; the paper's own limits. Each pane ends with key takeaways so you can stop after any section and still know what you learned.

point: GRPO drops the critic. For each prompt, sample G outputs from the current policy, compute the group mean and standard deviation of rewards, and use the standardized reward as the advantage. The crucial design choice for R1-Zero: use **rule-based rewards only** — no learned reward model. This sidesteps reward hacking that a learned RM would invite at this scale.

Rule-based reward is the sum of an accuracy term (correctness, verified by deterministic rules: math answer-in-a-box, code passing test cases) and a format term (encapsulating reasoning in `...` tags).

GRPO rule-based reward group-relative advantage

point: starting from DeepSeek-V3-Base with no SFT and only rule-based reward, R1-Zero's AIME 2024 pass@1 climbs from 15.6% to 77.9% over the RL run. With self-consistency @ 64 samples, 86.7%. Average response length grows from under 2K to over 6K tokens. The "aha moment" emerges mid-training without any human prompting — the model spontaneously re-evaluates its approach after hitting a wall.

Two emergent behaviors: (1) extended chain-of-thought as training progresses (response length grows monotonically), (2) reflective reasoning — backtracking, verification, alternative solution attempts. Both arise from pure RL; neither is in the training data.

AIME trajectory reflection / backtracking aha moment

point: R1 addresses R1-Zero's readability and language-mixing issues through four stages. Cold-start SFT on a small long-CoT set seeds high-quality reasoning patterns. Reasoning-oriented RL sharpens the math/code/logic reasoning. Rejection sampling builds a curated SFT corpus mixing reasoning and non-reasoning data. The second RL stage adds preference-based rewards and a language-consistency reward.

The language-consistency reward — Num(target-language words) / Num(total words) — pushes the model away from mixing Chinese and English mid-thought. The preference-based reward uses a learned RM for general-purpose queries (helpfulness + harmlessness) alongside the rule-based reward on math/code.

cold-start SFT reasoning RL rejection sampling language reward

point: 800K R1-generated samples (600K reasoning + 200K non-reasoning) suffice to transfer R1's reasoning capability to dense 1.5B–70B models via plain SFT. No RL at the smaller scale. Qwen2.5-7B distilled reaches 93.0% MATH-500 and 55.5% AIME 2024 — beating every open-source 7B reasoning model. Llama-3.3-70B distilled matches GPT-4-class performance.

Distillation works even at small scales: Qwen2.5-1.5B distilled reaches 28.9% AIME 2024 — far above Qwen2.5-1.5B Math-Instruct. The reasoning patterns transfer even when the distillation data is six orders of magnitude smaller than what RL would have required.

distilled model R1 teacher reasoning trace

point: R1 cannot emit structured output (JSON, function calls) or use external tools — its reasoning ceiling is tool-free. It produces long traces even on simple questions (overthinking). It occasionally mixes Chinese and English mid-CoT, especially for queries in non-supported languages. Few-shot prompts degrade its performance — the model is tuned for zero-shot. RL is expensive on long evaluation tasks; SWE-Bench at scale is open.

Rule-based rewards don't exist for every task — open-ended Q&A, creative writing, dialogue. Scaling pure RL to these domains requires either human annotation (expensive) or a robust reward model (susceptible to reward hacking). This is the open frontier.

limit open direction