Direct Preference Optimization: Your Language Model is Secretly a Reward Model

core: DPO bypasses reinforcement learning from human feedback by deriving the optimal RLHF policy in closed form. Fine-tuning becomes a binary classification loss on preference pairs — no separate reward model, no policy rollouts during training, no KL hyperparameter sweep beyond the implicit temperature β. The implicit reward formulation exposes what DPO actually does at the gradient level: it increases the likelihood of the preferred completion and decreases the likelihood of the dispreferred one, weighted by how confidently the implicit reward model rates the chosen pair.

Five panes walk the math derivation step by step: the Bradley-Terry preference model, the KL-constrained reward-maximization objective, its closed-form optimal policy, the reparameterized reward, and the resulting DPO loss. The fourth pane prints the win-rate tables verbatim. The fifth pane lists the paper's own limits.

primer — what you need before you start

the task. DPO targets alignment — making a language model prefer the responses humans prefer. Training data is (chosen, rejected) pairs of completions; the model is updated so the chosen becomes more likely than the rejected. This replaces the older RLHF pipeline of fitting a reward model then running PPO.

how it was done before. Two stages: (1) train a separate reward model on pairwise preferences (Bradley-Terry); (2) run PPO with on-policy rollouts against the reward model, KL-pinned to a reference SFT model. Stable only with careful KL tuning, value-head training, and a lot of compute. Single-stage alternatives like DPO exist because the two-stage pipeline is fragile.

glossary — terms you will see.
SFT model — supervised fine-tuned base model; the starting point for both RLHF and DPO.
reference model $\\pi_{\\text{ref}}$ — a frozen copy of the SFT model; DPO KL-pins against this.
reward model — a separate model trained on pairwise preferences (RLHF only; DPO eliminates this).
PPO — Proximal Policy Optimization; on-policy RL that runs rollouts during training.
Bradley-Terry (BT) model — $P(A \\succ B) = \\sigma(r(A) - r(B))$; turns pairwise comparisons into scalar rewards.
KL constraint — bound on how far the trained policy can drift from the reference. In RLHF it's a hyperparameter; in DPO it's baked into the loss via $\\beta$.
$\\beta$ — temperature that controls how sharp the preference signal is. Higher $\\beta$ = stricter KL to reference.
implicit reward — DPO's reward function $\\hat{r}(x,y)$ is never trained; it's defined by the policy ratio to reference.

symbol table — what the letters mean.
\\pi_\\theta(y\\vert x) — the policy: probability of completion y given prompt x.
$\\pi_{\\text{ref}}(y\\vert x)$ — frozen reference (SFT) policy.
\\hat{r}_\\theta(x, y) — DPO's implicit reward: \\beta \\log \\frac{\\pi_\\theta(y\\vert x)}{\\pi_{\\text{ref}}(y\\vert x)} + Z(x).
$\\beta$ — temperature (KL sharpness).
$(x, y_w, y_l)$ — one preference pair: prompt x, chosen (winner) $y_w$, rejected (loser) $y_l$.
$\\mathcal{L}_{\\text{DPO}}$ — the binary classification loss on log-ratios.
$Z(x)$ — partition function; cancels in pairwise comparisons so it never needs to be computed.

tone legend — what the [brackets] mean.
[rlhf] RLHF / PPO   [deriv] derivation   [obs] observation
[bt] Bradley-Terry   [policy] optimal policy   [loss] DPO loss
[exp] experiment result   [limit] paper-reported limit
[take] key takeaway   [sys] system / meta

Five panes cover, in pedagogical order: RLHF pipeline problem, Bradley-Terry preference model + KL-constrained objective, closed-form optimal policy, the reparameterized DPO loss, and the paper's own limits. Each pane ends with key takeaways so you can stop after any section and still know what you learned.

point: the standard recipe has three stages. SFT warms up the policy on demonstration data. Reward model $r_\phi$ fits pairwise preferences with a Bradley-Terry classification loss. RL runs PPO on rollouts from the current policy, using $r_\phi$ as the reward and a KL penalty against the SFT reference policy. PPO needs a value head, clip ratios, advantage estimates, and on-policy sampling every gradient step.

SFT model π_ref is the starting point for all three stages. The reward model and the policy are trained separately; the policy gradient flows through the reward model, not through the preference data directly.

SFT stage reward model fit PPO rollout KL penalty

point: preferences over pairs of completions are modeled by a Bradley-Terry distribution over their (latent) scalar rewards. The preference probability is a softmax of the difference: the higher-rated completion wins with probability equal to the sigmoid of the reward gap.

A single preference pair $(x, y_1, y_2)$ with $y_1 \succ y_2$ contributes a binary log-likelihood of the sigmoid. The reward function $r^*$ is never observed — it is the unknown signal the preference data labels.

preference model softmax form

point: three moves turn RLHF into DPO. First, the closed-form optimal policy of the KL-constrained reward-maximization objective (proportional to π_ref · exp(r/β)). Second, invert that to express the reward as a log-ratio of policy to reference. Third, substitute the reward into the Bradley-Terry preference model and take the maximum-likelihood objective. The result is a single classification loss — no sampling, no value head, no separate reward network.

The gradient of the DPO loss has an interpretable form: it nudges up the log-likelihood of the preferred completion and down the log-likelihood of the dispreferred one, scaled by how confidently the implicit reward model rates the chosen pair.

optimal policy implicit reward DPO loss gradient update

point: three open-ended text-generation tasks. IMDb sentiment: DPO traces the highest expected reward-vs-KL frontier, beating even PPO with access to the ground-truth sentiment classifier. TL;DR summarization: 61% win rate at the optimal sampling temperature, versus 57% for the best PPO. Anthropic-HH dialogue: DPO is the only method that improves over the chosen-summaries baseline.

All numbers are paper-report: GPT-4 is used as the win-rate evaluator, and a separate human study confirms GPT-4 rankings correlate with human rankings as much as humans correlate with each other.

DPO PPO SFT / Preferred-FT Best of N baseline

point: DPO depends on a fixed offline preference dataset and cannot do on-policy exploration the way PPO can. Out-of-distribution generalization is comparable to PPO, not better. The win-rate evaluation itself relies on GPT-4 as a proxy for human preference — the paper's own human study shows GPT-4 is a noisy per-sample proxy even when it correlates well in aggregate. The paper evaluates up to 6B parameters; scaling to larger LMs is an open direction.

The same offline-preference dependency makes self-labeling rollouts impossible without an external reward model — the direction taken by RLOO and Self-Rewarding LM uses the policy's own samples but still needs a learned reward signal.

limit open direction