Qwen2.5 Technical Report

core: Qwen 2.5 is a comprehensive LLM family from Alibaba — seven open-weight dense models (0.5B / 1.5B / 3B / 7B / 14B / 32B / 72B) plus the proprietary Qwen2.5-Turbo (MoE with 1M-token context) and Qwen2.5-Plus. Pre-training scales to 18T tokens; post-training is SFT + offline DPO + online GRPO.

Five panes: data pipeline, post-training, long context, multilingual and coding, and the paper's own limits.

primer — what you need before you start

the task. Qwen 2.5 is a comprehensive open-weight LLM family from Alibaba — the largest open alternative to Llama 3 as of 2024-2025. The paper documents pre-training (18T tokens), post-training (SFT + DPO + GRPO), long-context extension (128K / 1M), multilingual coverage (29+ languages), and coding ability (HumanEval 86.6).

how it was done before. Qwen 2 (2023) was a solid mid-tier open model. Qwen 2.5 doubled the data, expanded the post-training pipeline, and added long-context — making it competitive with Llama 3 70B / 405B on most benchmarks.

glossary — terms you will see.
Qwen 2.5 — the open-weight family (0.5B / 1.5B / 3B / 7B / 14B / 32B / 72B).
Qwen2.5-Turbo — proprietary MoE variant with 1M-token context.
SFT — Supervised Fine-Tuning; 1M+ examples.
DPO — Direct Preference Optimization; ~150K preference pairs.
GRPO — Group Relative Policy Optimization; on-policy RL with rule-based rewards.
YaRN — position-interpolation extension to 128K context.
InfLLM — training-free long-context via segment-level sparse attention.
dual chunk attention — split long context into chunks; query attends within and across chunks.

symbol table — what the letters mean.
$\pi_\theta(y \vert x)$ — the policy.
$\pi_{\text{ref}}$ — the reference SFT model.
$\hat{r}(x, y)$ — implicit reward: $\beta \log \frac{\pi_\theta(y \vert x)}{\pi_{\text{ref}}(y \vert x)}$.
$\mathcal{L}_{\text{GRPO}}$ — the GRPO loss; no critic, advantage = group mean.
$\beta$ — temperature / KL sharpness.
$\mathcal{D}$ — the 18T-token training corpus.
$\theta$ — the model weights (what gets trained).

tone legend — what the [brackets] mean.
[obs] observation   [pipe] data pipeline
[sft] SFT step   [ctx] long context
[eq] equation   [take] key takeaway
[sys] system / meta   [limit] paper-reported limit

Six panes cover, in pedagogical order: data pipeline; post-training; long context; multilingual and coding; open directions. Each pane ends with key takeaways so you can stop after any section and still know what you learned.

point: 18T tokens curated through five stages. Knowledge-base upgrade via fastText classifier on Common Crawl and open sources, three-level deduplication (URL / document / line), heuristic filtering, domain classification, and strategic mixing that upsamples high-quality domains while downsampling over-represented content like e-commerce.

Final mix: 56% Common Crawl + 4% AlgebraicStack + 10% arXiv + 20% GitHub code + 10% natural-language. Slightly more English than non-English in the base mix; the multilingual expert bootstraps the non-English side.

web + dedup math / code multilingual domain mixing

point: three stages. SFT on over 1M examples with system-prompt diversity, automatic response-quality scoring by a critic + multi-agent system, and cross-lingual instruction translation. Offline DPO resamples responses for the same query, marking failed as negative and successful as positive, training on ~150K pairs. Online GRPO samples 8 responses per query, uses response-score variance as the baseline.

Five labeling criteria for the online RL: truthfulness, helpfulness, conciseness, relevance, harmlessness, debiasing. 2,048 global batch with 2,048 samples per episode. The Online Merging Optimizer handles DPO optimization.

SFT offline DPO online GRPO 5 labeling criteria

point: Qwen2.5 extends to 128K for dense models (0.5B–72B) and to 1M for the Qwen2.5-Turbo MoE variant. The combination of techniques: YaRN for RoPE frequency-dependent scaling, Dual Chunk Attention (DCA) to separate very long inputs from short, and InfLLM for training-free long-context inference.

RoPE base frequency is fixed at 10K, 1M, or 500K depending on the variant. 4D parallelism (TP × CP × PP × DP) handles the long sequences at training time; sequence parallelism keeps the context parallel across devices.

YaRN DCA InfLLM

point: 29+ languages supported via cross-lingual instruction translation in SFT. Code in 40+ languages with GitHub-sourced pre-training (~20% of the data mix). Qwen2.5-72B-Instruct: 86.6% HumanEval, 84.7% MBPP, 55.5% LiveCodeBench.

Multilingual capabilities: Multi-Exam 78.7% / Multi-Understanding 89.6% / Multi-Mathematics 76.7% / Multi-Translation 39.0%. Code generation is the closest to GPT-4o among open-weight models at the time of release.

multilingual code high-resource

point: Qwen 2.5 still trails o1 on the hardest reasoning tasks. Hallucinations remain on closed-domain factual queries. Multilingual coverage is uneven — high-resource Latin / Germanic languages far better than African / South Asian. Even with YaRN, reasoning quality on multi-hop 128K+ contexts degrades. Tool use and multi-step agents are competitive but behind Claude 3.5 Sonnet / GPT-4o.

Pre-training has a temporal cutoff — no live web access without a separate RAG pipeline. The Online Merging Optimizer is a new technique; longer-horizon stability is not fully characterized.

limit open direction