The Math of DPO, Explained

From RLHF to a single loss

Aligning a model from human preferences was, for a while, a three-stage pipeline: collect preference data, train a reward model to score responses, then optimize the language model against that reward with reinforcement learning, usually PPO, while a penalty component keeps it from drifting too far from where it started. DPO's contribution is to show that for this particular objective the reward model and the RL are unnecessary: the whole thing collapses into one supervised loss you can train the same way you train anything else. To see why, it helps to start from the objective the RL stage was solving.

Optimization Objective

The RL stage maximizes the expected reward of the model's responses while paying a penalty, controlled by $\beta$, for moving away from a reference policy$\;\pi_{\text{ref}}$, typically the initial supervised-fine-tuned model you begin with. The penalty is a KL divergence, and it is what stops the model from collapsing onto whatever degenerate output the reward happens to score highest. So, in simple terms, the goal is to maximize the reward signal without drifting too far away from the starting point.

This objective has a known closed-form solution. For a fixed reward, the policy that maximizes reward minus that KL penalty is the reference policy reweighted by the exponentiated reward, a Boltzmann distribution over responses:

The obstacle here is $Z(x)$, the normalizing constant. It sums over every possible response $y$, of which there are astronomically many, so it cannot be computed. This partition function is exactly what makes the closed-form solution look useless in practice, but the DPO approach makes this term disappear.

Rewriting the reward in terms of the policy

The first move is to turn the optimal-policy equation around. Instead of writing the policy in terms of the reward, solve for the reward in terms of the policy. Taking logs of both sides and rearranging gives the reward as a scaled log-ratio between the optimal policy and the reference, plus a term that depends only on the prompt:

This is the sense in which, as the paper's title puts it, your language model is secretly a reward model: any policy implicitly defines a reward, recoverable from how far it has moved from the reference. The difficult term $\beta\log Z(x)$ is still there, but notice that it depends only on the prompt $x$, not on the response $y$.

Why the normalizing term disappears

Preferences are modeled with Bradley–Terry: the probability that a labeler prefers response$\;y_w$ over $\;y_l$ is the sigmoid of the difference in their rewards. It depends on the two rewards only through their difference, never their absolute values.

Now substitute the reward expression into that difference. Both responses share the same prompt, so both carry the same $\beta\log Z(x)$ term, which cancels out when computing difference. The uncomputable component is gone, and what remains is written entirely in quantities you can evaluate: log-probabilities under the policy and under the reference.

The DPO loss

The last step is to stop treating $\pi^{*}$ as an unknown optimum and simply make it the model we train, $\pi_\theta$. Fitting the Bradley–Terry probability above to the observed preferences by maximum likelihood or equivalently, minimizing its negative log-likelihood, gives you the DPO loss.

The scaled log-ratio between policy and reference is the implicit reward the model assigns to a response:

Read that way, the loss is exactly that of a binary classifier: it asks the implicit reward to rank the chosen response above the rejected one, and its shape is the familiar logistic loss on the margin between the two.

What the gradient does

Writing $\text{margin} = \hat{r}_\theta(x,y_w) - \hat{r}_\theta(x,y_l)$, differentiating the loss gives a familiar-looking update with one important weight in front:

The part in the second bracket is intuitive: raise the log-probability of the chosen response, lower the log-probability of the rejected one. The part in front is interesting:$\;\sigma(\hat{r}_l - \hat{r}_w) = \sigma(-\text{margin})$ is large when the model has the pair backwards (when it currently assigns more implicit reward to the rejected response) and small when the model already ranks the pair correctly by a wide margin. It's like putting a dynamic weight on the gradient. We can watch that weight behave as the margin grows.

import numpy as np

def sigmoid(z):      return 1.0 / (1.0 + np.exp(-z))
def log_sigmoid(z):  return -np.logaddexp(0.0, -z)          # stable log(sigmoid(z))

def dpo_loss(logp_chosen, logp_rejected, ref_chosen, ref_rejected, beta):
    """DPO loss for one preference pair, from sequence log-probs under policy and reference."""
    margin = beta * ((logp_chosen - ref_chosen) - (logp_rejected - ref_rejected))
    return -log_sigmoid(margin)

print("margin   loss     gradient weight = sigmoid(-margin)")
for margin in [-4, -1, 0, 1, 4, 8]:
    print(f" {margin:+d}    {-log_sigmoid(margin):6.4f}   {sigmoid(-margin):.4f}")

margin   loss     gradient weight = sigmoid(-margin)
 -4    4.0181   0.9820
 -1    1.3133   0.7311
 +0    0.6931   0.5000
 +1    0.3133   0.2689
 +4    0.0181   0.0180
 +8    0.0003   0.0003

At a negative margin (the model preferring the rejected answer) the loss is large and the weight is near one, so the update is strong. As the margin grows positive, both the loss and the weight fall toward zero: once the model confidently prefers the chosen answer, DPO stops pushing. This self-limiting behavior is why the reference and $\beta$ matter.

Checking the gradient

The analytic gradient with respect to the chosen response's log-probability is$\;-\beta\,\sigma(-\text{margin})$, and with respect to the rejected one it is the same magnitude with the opposite sign. A finite-difference check confirms it, and the same run shows the identity that the implicit-reward margin equals the DPO margin exactly.

beta = 0.1
logp_chosen, logp_rejected, ref_chosen, ref_rejected = -2.0, -3.0, -2.5, -2.5
margin = beta * ((logp_chosen - ref_chosen) - (logp_rejected - ref_rejected))
weight = sigmoid(-margin)

grad_chosen   = -beta * weight            # analytic gradients wrt the policy log-probs
grad_rejected = +beta * weight

eps = 1e-6                                 # finite-difference check on the chosen side
fd_chosen = (dpo_loss(logp_chosen+eps, logp_rejected, ref_chosen, ref_rejected, beta)
             - dpo_loss(logp_chosen-eps, logp_rejected, ref_chosen, ref_rejected, beta)) / (2*eps)
print(f"grad wrt chosen  : analytic {grad_chosen:+.6f}   finite-diff {fd_chosen:+.6f}")
print(f"grad wrt rejected: analytic {grad_rejected:+.6f}   finite-diff {(-fd_chosen):+.6f}")

r_hat_chosen   = beta * (logp_chosen   - ref_chosen)     # implicit reward = beta * log(pi / pi_ref)
r_hat_rejected = beta * (logp_rejected - ref_rejected)
print(f"implicit-reward margin  r_hat_chosen - r_hat_rejected = "
      f"{r_hat_chosen - r_hat_rejected:+.4f}  (== DPO margin {margin:+.4f})")

grad wrt chosen  : analytic -0.047502   finite-diff -0.047502
grad wrt rejected: analytic +0.047502   finite-diff +0.047502
implicit-reward margin  r_hat_chosen - r_hat_rejected = +0.1000  (== DPO margin +0.1000)

Watching it learn

We don't need an actual transforme to inspect the mechanics of this. Collapse the model down to a single categorical distribution over a handful of tokens (a one-step policy $\pi_\theta(y)=\mathrm{softmax}(\theta)_y$ ), fix a reference, and give it one preference: token “chosen” beats token “rejected.” Running gradient descent on the DPO loss does exactly what we expect from the formula. The chosen probability climbs, the rejected one falls, the margin grows, and because the update is scaled by that self-limiting weight, each step gets smaller as the model becomes more confident.

def log_softmax(x): return x - np.logaddexp.reduce(x)
def softmax(x):     e = np.exp(x - x.max()); return e / e.sum()

vocab_size = 5
rng = np.random.default_rng(0)
ref_logits = rng.normal(0, 1, vocab_size)
theta = ref_logits.copy()                 # start the policy at the reference
chosen, rejected = 0, 1
beta, learning_rate = 0.1, 0.5
ref_logp = log_softmax(ref_logits)

print("step   pi(chosen)  pi(rejected)   margin")
for step in range(601):
    logp = log_softmax(theta)
    margin = beta * ((logp[chosen] - ref_logp[chosen]) - (logp[rejected] - ref_logp[rejected]))
    weight = sigmoid(-margin)                       # adaptive: small once the model is confident
    grad = np.zeros(vocab_size)
    grad[chosen]   = -weight * beta
    grad[rejected] = +weight * beta
    if step in (0, 50, 150, 300, 600):
        pi = softmax(theta)
        print(f"{step:4d}     {pi[chosen]:.4f}      {pi[rejected]:.4f}     {margin:+.4f}")
    theta -= learning_rate * grad

step   pi(chosen)  pi(rejected)   margin
   0     0.2024      0.1564     +0.0000
  50     0.4877      0.0358     +0.2353
 150     0.8783      0.0013     +0.6280
 300     0.9855      0.0000     +1.0746
 600     0.9993      0.0000     +1.6737

The policy starts exactly at the reference and walks away from it in the one direction the preference asks for. This is a miniature DPO : no reward model, no rollouts, just log-probabilities under two models and a logistic loss between them.

What $\beta$ controls

The temperature $\beta$ is the same one from the KL penalty, and it sets the exchange rate between preference and drift. Because the margin is$\;\beta$ times the difference in log-ratios, reaching a given preference strength requires a log-ratio gap of margin divided by $\beta$. To make the model prefer the chosen answer with probability $0.9$, a small $\beta=0.1$ demands a log-ratio gap near $22$, while $\beta=0.5$ reaches the same preference with a gap near $4.4$. A larger $\beta$ buys the same preference for a smaller move away from the reference; a smaller one lets the model travel further to satisfy the data. That is often a parameter you actually tune.

Common questions

Does DPO need a reward model?

No. The reward is never trained as a separate model; it is defined implicitly by the policy through the scaled log-ratio to the reference, and the derivation shows the preference probability can be written using only that. You still need preference data (chosen and rejected pairs) but not a reward network and not a reinforcement-learning loop.

Does DPO still use a reference model?

Yes. The reference $\pi_{\text{ref}}$ appears in every term through the log-ratio, so during training you evaluate log-probabilities under both the model being trained and a frozen reference, usually the supervised-fine-tuned checkpoint you started from. The reference is what the KL penalty is measured against, and it is what keeps $\beta$ meaningful.

How is DPO different from RLHF with PPO?

They optimize the same KL-regularized objective; they differ in how. RLHF fits an explicit reward model and then improves the policy against it with reinforcement learning. DPO uses the closed-form relationship between the optimal policy and the reward to skip both, turning the problem into a single supervised loss on preference pairs. The trade is simplicity and stability for the flexibility of having a standalone reward you can reuse or inspect.

References

• Rafailov, Sharma, Mitchell, Ermon, Manning & Finn, “Direct Preference Optimization: Your Language Model is Secretly a Reward Model” (2023) — the derivation, the loss, and the gradient reproduced here.
• Bradley & Terry, “Rank Analysis of Incomplete Block Designs” (1952) — the preference model whose reward difference makes the normalizer cancel.

The NumPy here is meant to illustrate the mathematics on a CPU, not to fine-tune a real model — in practice the log-probabilities are summed over the tokens of full sequences under a transformer, but the loss, the gradient, and the implicit-reward identity are exactly the ones shown.