The Llama 3 Herd of Models

core: Llama 3 is a herd of multilingual dense Transformers (8B / 70B / 405B) trained on 15.6T tokens with a 128K context window. The post-trained 405B-Instruct closes the gap to GPT-4 on most benchmarks and approaches Gemini-Ultra on instruction following.

Five panes: data pipeline, post-training recipe, multi-modal composition, long context, and the paper's own limits.

primer — what you need before you start

the task. Llama 3 is a flagship open-source LLM family from Meta. The paper documents the entire training pipeline: data, architecture, post-training, multi-modal, long context. The goal is to be competitive with closed frontier models while staying fully open.

how it was done before. Llama 2 (2023) was solid but a generation behind GPT-4. The Llama 3 bet: more data, more math/code, longer context, and a heavy post-training pipeline. The result closes most of the gap to GPT-4 in 2024.

glossary — terms you will see.
token — a chunk of text the model reads; Llama 3 uses BPE on a 128K vocab.
embedding — the vector each token turns into at the input layer.
d_model — embedding dim: 4096 (8B) / 8192 (70B) / 16384 (405B).
RoPE — Rotary Position Embedding; the position encoding scheme.
DPO — Direct Preference Optimization; the alignment loss used in the post-training pipeline.
RM — Reward Model; trained on pairwise preferences, used for rejection sampling.
RS — Rejection Sampling; keep the top-k highest-reward completions, fine-tune on them.
multi-modal — the LM processes text only; images / audio / video are encoded separately and fed in.

symbol table — what the letters mean.
$\pi_\theta(y \vert x)$ — the policy: probability of completion y given prompt x.
$\pi_{\text{ref}}$ — the reference SFT model; KL anchor for post-training.
$\hat{r}(x, y)$ — implicit reward: $\beta \log \frac{\pi_\theta(y \vert x)}{\pi_{\text{ref}}(y \vert x)} + Z(x)$.
$\beta$ — temperature / KL sharpness.
$\theta$ — the model weights (what gets trained).
$\mathcal{D}$ — the training dataset of curated tokens.

tone legend — what the [brackets] mean.
[obs] observation   [pipe] data pipeline
[base] base model   [dpo] DPO / alignment step
[mm] multi-modal   [rs] rejection sampling
[eq] equation   [take] key takeaway
[sys] system / meta   [limit] paper-reported limit

Six panes cover, in pedagogical order: data pipeline; post-training recipe; multi-modal composition; long-context extension; open directions. Each pane ends with key takeaways so you can stop after any section and still know what you learned.

point: 15.6T tokens curated from Common Crawl and other sources. URL / document / line-level dedup, heuristic filtering (n-gram ratio, dirty-word count, KL divergence), then model-based quality classifiers — fastText reference (Wikipedia), DistilRoBERTa for general quality.

Final mix: roughly 50% general knowledge, 25% math and reasoning, 17% code, 8% multilingual. Domain-specific filtering targets math (LaTeX / MathJax), code (GitHub repos), and multilingual (parallel texts).

web math / code multilingual quality filter

point: six rounds of alternating reward modeling, rejection sampling, SFT, and DPO. The same recipe used for Llama 2 but at much larger scale and with three specific modifications to DPO — mask formatting tokens from the loss, add NLL regularization on the chosen sequence, increase batch size.

DPO is run on the same preference data the reward model trains on. This keeps the distribution aligned. Rejection sampling uses K = 10–30 outputs from the latest checkpoint; the best candidate is picked by reward model (capability) or by human preference (code / reasoning / safety).

rejection sampling DPO SFT model averaging

point: instead of retraining the LM, Meta composes vision and speech on top of the frozen Llama 3 405B text backbone. A ViT image encoder (400M params) is trained from scratch on 6B image-text pairs with text-description generation as the objective. Cross-attention vision adapters are trained on 2.5M image-instruction pairs.

Video adapter aggregates information across frames. Speech encoder is Whisper-style trained with masked reconstruction on ASR and translation data. The LM weights stay frozen throughout — only the adapters are trained.

image encoder video adapter speech encoder cross-attention adapter

point: RoPE base raised to 500,000 — supports context extension up to 128K without retraining from scratch. Long-context pre-training in the final stages uses ~800B tokens to extend the supported context length incrementally.

Long-context performance validates the extension: QuALITY 56.0 → 87.6 from 8B to 405B; GSM8K 16-shot 60.0 → 90.0. The 405B model passes needle-in-a-haystack retrieval across the full 128K window.

RoPE long-context tokens needle-in-haystack

point: Llama 3 405B closes the gap to GPT-4 on most benchmarks but trails GPT-4o and Claude 3.5 Sonnet on the hardest coding and reasoning tasks. Hallucinations remain, especially on closed-domain factual queries. Multilingual coverage is uneven — high-resource Latin / Germanic languages far better than low-resource African / South Asian.

3.8 × 10²⁵ FLOPs is compute-optimal at 405B; the 8B and 70B are undertrained by the same scaling law, trading efficiency for inference cost. The compute recipe does not directly translate to open-science replication.

limit open direction