Training Compute-Optimal Large Language Models

core: for a fixed training compute budget, current large language models are significantly undertrained — their size was scaled up while the number of training tokens stayed roughly constant. Training over 400 models, Hoffmann et al. find the compute-optimal frontier scales parameters and tokens equally. The test model, Chinchilla (70B parameters, 1.4T tokens, same compute as Gopher), beats Gopher (280B), GPT-3 (175B) and MT-NLG (530B).

Five panes: the undertraining problem, the three ways to find the frontier, the 400-model experiment and Chinchilla itself, the downstream results, and the paper's own limits plus a 2026 reality check.

primer — what you need before you start

the task. Training a language model costs compute. Given a fixed compute budget, you must decide how to split it between making the model bigger (more parameters) and showing it more data (more training tokens). This paper answers which split gives the lowest final loss.

how it was done before. Kaplan et al. (2020) fitted scaling laws and concluded you should grow the model much faster than the data. That advice made GPT-3 (175B params, ~300B tokens) and Gopher (280B params, 300B tokens) — big models trained on relatively little data. This paper revisits that conclusion with a cleaner experimental design and finds the opposite.

glossary — terms you will see.
parameters (N) — the model's weights; how "big" it is.
tokens (D) — how much text the model was trained on.
compute (C) — total FLOPs spent training; roughly $C \approx 6ND$.
loss — how surprised the model is by the next token (cross-entropy); lower means better predictions. This is the number every scaling law here fits.
compute-optimal — the N / D split that gives the lowest loss for a given C.
IsoFLOP — fixing compute and sweeping model size to find the best loss.
undertrained — a model too large for the amount of data it saw.
MMLU — a 57-subject knowledge benchmark used to compare models.

symbol table — what the letters mean.
$N$ — number of parameters.
$D$ — number of training tokens.
$C$ — training compute, $C \approx 6ND$.
$L(N, D)$ — final validation cross-entropy loss; fitted as a floor $E$ plus one term falling with $N$ and one falling with $D$ (full form in the approaches pane).
$E$ — irreducible entropy: the loss floor no model can pass.

tone legend — what the [brackets] mean.
[obs] observation / setup   [app] an approach to the frontier
[law] scaling law / formula   [tbl] experiment / table numbers
[res] headline result   [limit] limitation
[eq] equation   [take] key takeaway
[sys] system / meta   [2026] site-editorial reality check

Six panes cover, in pedagogical order: the undertraining problem; three independent ways to locate the compute-optimal frontier; the 400-model experiment and Chinchilla itself; the downstream results; and the paper's limits with a 2026 reality check. Each pane ends with key takeaways so you can stop after any section and still know what you learned.

point: the era's dominant recipe scaled parameters and held training data roughly constant. The result: current large models are significantly undertrained — too big for the data they saw. The paper's framing: given a compute budget, the size and the number of tokens are a trade-off, and everyone was on the wrong side of it.

The diagnosis is not about quality of data or architecture. It is about the allocation of a fixed compute budget between two quantities — model size and training tokens — and that prior work had systematically over-spent on the first.

observation the trade-off the diagnosis

point: the authors locate the compute-optimal frontier three independent ways. Approach 1 fixes model sizes and varies tokens, taking the lowest-loss envelope. Approach 2 is IsoFLOP: fix compute, sweep model size, keep the lowest loss. Approach 3 fits a parametric loss function of parameters and tokens and optimizes it under a FLOP constraint. All three agree.

The three methods are deliberately different — an envelope of runs, a set of fixed-compute sweeps, and a closed-form fit — so their agreement is strong evidence, not an artifact of one method.

an approach fitted law the conclusion

point: over 400 models, from 70 million to over 16 billion parameters, trained on 5 to 500 billion tokens, map out the loss surface. At Gopher's compute budget — the same number of FLOPs — the compute-optimal model is Chinchilla: 70B parameters on 1.4T tokens. The recipe generalizes: roughly 20 tokens per parameter.

The compute-optimal pairings show that as compute grows, both parameters and tokens grow together, with tokens slightly outpacing parameters in raw count.

table / data setup the law

point: the equal-scaling prediction holds. At one quarter of Gopher's parameters, Chinchilla uniformly and significantly outperforms Gopher (280B), GPT-3 (175B), Jurassic-1 (178B), and Megatron-Turing NLG (530B) across downstream evaluation tasks — while using far less compute for fine-tuning and inference. MMLU: 67.6% against Gopher's 60.0%.

The payoff is not just accuracy. A smaller, better-trained model is cheaper to serve: less memory, less latency, less inference cost for the same or better quality.

Chinchilla baselines numbers

point: the paper is honest about its footing. Only two comparable large-scale runs (Gopher, Chinchilla) validate the laws. The frontier is assumed to be a power law, though the authors note some concavity. And the scaling runs all trained on less than one epoch of data.

The 2026 reality check that follows is this site's own editorial note, added at the request of the site owner — it is analysis written in 2026, not a claim from Hoffmann et al.

limit 2026 reality check