Vaswani et al. (2017) replaced the recurrent encoder-decoder stack with a pure attention architecture: six encoder layers and six decoder layers, each containing multi-head self-attention, position-wise feed-forward networks, residual connections, and layer normalization. The single architectural change — removing recurrence entirely — let the model parallelize across sequence positions during training (replacing the recurrent layer’s sequential operations over the sequence with self-attention’s constant), shortened the maximum path length to constant for any two positions, and made long-range dependencies directly learnable (the encoder self-attention example showing how a verb like “making” attends to a distant dependent to complete the phrase “making…more difficult”). With base hyperparameters (6 layers, d_model=512, h=8 heads, d_ff=2048) the model trains in roughly 12 hours on 8 P100 GPUs and reaches 27.3 BLEU on WMT 2014 English-to-German; the big variant (d_model=1024, h=16, 300K steps) hits 28.4 BLEU on English-to-German and 41.8 on English-to-French — state of the art in 2017 with a small fraction of the prior models’ training cost.
The Tensor2Tensor reference implementation has been superseded by HuggingFace transformers in practice, but the original scaffold — scaled dot-product attention, multi-head projection, sinusoidal position encoding, warmup-then-inverse-square-root learning rate, label smoothing — has survived a decade of variants. The 2024–2026 frontier (FlashAttention, RoPE, MQA / GQA, MLA, MoE, sliding-window attention, linear-time alternatives like Mamba) modifies the attention pattern, position encoding, and routing but keeps the same Q / K / V + residual + LayerNorm scaffold. The ablation in Table 3 (sections A through E) shows that single-head attention underperforms multi-head, reducing the per-head dimension hurts, bigger models monotonically improve, and sinusoidal position encoding is essentially equivalent to learned. Trajectories are schematic; numbers are paper-report. This site does not host the PDF.
teaching explainer · Vaswani et al. · NeurIPS 2017 · arXiv:1706.03762 · Google Brain / Google Research · arXiv:1706.03762 | code tensorflow/tensor2tensor (archived 2023-06, superseded by huggingface/transformers) | traces are schematic; numbers marked paper-report come from paper tables
0overview + primer — start here
core: a single architectural change — replacing recurrence with pure multi-head self-attention — gave 28.4 BLEU on WMT 2014 EN-DE and 41.8 on EN-FR, at roughly one-tenth the training cost of prior state-of-the-art encoder-decoders. The scaffold (Q/K/V attention + residual + LayerNorm + sinusoidal position encoding) has survived a decade of variants and underlies every modern LLM.
6 + 6 layers
N stacked encoder + decoder layers, each with multi-head self-attention, position-wise feed-forward, residual connections, and layer normalization. No recurrence, no convolution.
Q KT / √dk · V
Softmax of (Q · KT / √dk) multiplied by V. Scaling by √dk prevents softmax saturation when dk is large.
O(1) path length
Self-attention connects any two positions with O(1) operations, vs O(n) for recurrent and O(logk(n)) for convolutional stacks. Long-range dependencies are directly learnable.
primer — what you need before you start
the task. Translation takes a sequence of input words and produces a sequence of output words. Example: English "the cat sat on the mat" → French "le chat est assis sur le tapis". A neural model reads the input and writes the output — token by token, each conditioned on what came before.
how it was done before. Until 2017, most translation models were RNNs — they read the input word-by-word, left-to-right. The problem: by the time the model finishes reading, the early words have to "remember" everything through a chain of updates, which loses information on long sentences. Training is also slow because each word depends on the previous one. CNNs are parallel but have their own problem: each layer only "sees" a fixed-size window, so a long-range dependency (verb 30 words back) requires many stacked layers to traverse.
what changed in 2017. Vaswani et al. replaced recurrence with an operation that lets every word look directly at every other word, in one step. That's attention. The rest of this demo explains how it works.
glossary — terms you will see.
token — a chunk of text the model reads or writes (a word, subword, or character).
embedding — turning a token into a fixed-size vector of numbers so a neural net can do math with it.
d_model — the size of those embedding vectors. base Transformer uses 512.
softmax — a function that turns any list of real numbers into a probability distribution (all values between 0 and 1, summing to 1). Used to make "which token should I write next?" decisions.
BLEU — score from 0 to 100 measuring how close a machine translation is to a human reference. Higher is better.
PPL (perplexity) — how confused the model is. Lower is better.
logit — the raw score the model outputs BEFORE softmax turns it into probabilities.
encoder — the part of the model that reads the input sentence and builds a representation of it.
decoder — the part that writes the output sentence one token at a time, looking back at the encoder's representation.
attention — a mechanism that lets the decoder (or any layer) decide which input positions to look at when producing each output. The entire paper is about how to do this well.
symbol table — what the letters mean.
Q (query) — "what I'm looking for". One per output position.
K (key) — "what I can offer". One per input position.
V (value) — "the actual content". One per input position.
QKT — similarity matrix: how much each Q matches each K.
softmax(...) — converts the similarity matrix into a probability distribution.
dk — the dimension of the Q and K vectors (head_dim). base uses 64.
√dk — a scaling factor that prevents softmax from saturating.
WQ, WK, WV — three learnable projection matrices that produce Q, K, V from the input.
WO — the learnable projection that combines multi-head outputs.
h — the number of attention heads running in parallel. base uses 8.
tone legend — what the [brackets] mean.
[arch] architecture claim [obs] observation [eq] equation
[qkv] Q/K/V intuition [form] derivation [take] key takeaway
[abl] ablation result [fron] 2026 frontier [sys] system / meta
[recap] recapitulation [warn] caveat
Thirteen panes cover, in pedagogical order: complexity (why transformer beats RNN/CNN), Q/K/V intuition, scaled dot-product equation, multi-head, full architecture, positional encoding, position-wise feed-forward, residual + LayerNorm + embeddings, training (LR schedule, Adam, regularization), WMT 2014 BLEU + parsing results, six ablation dimensions, and the 2026 frontier (FlashAttention, RoPE, MQA / GQA, MLA, MoE, Mamba). Each pane ends with key takeaways so you can stop after any section and still know what you learned.
1complexity & path length — why transformer
2Q/K/V intuition — library & nearest-neighbor
3scaled dot-product attention — equation (1)
4multi-head attention — h parallel heads
5architecture — encoder-decoder stack
6positional encoding
7position-wise feed-forward — equation (2)
8residual + LayerNorm + embeddings
9training — LR schedule, Adam, regularization
10results — BLEU + parsing
11ablations — six dimensions
122026 frontier — variants & alternatives
Schematic traces are teaching schematics. Numbers reported: Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, Polosukhin, Attention Is All You Need, NeurIPS 2017 · arXiv:1706.03762v7 (2 Aug 2023), arXiv:1706.03762. Reference impl: tensorflow/tensor2tensor (17.4k stars, archived 2023-06). Modern fork: huggingface/transformers. This site does not host the PDF.