Skip to content
Agent YAP

What is it?

DPO

DPO stands for Direct Preference Optimization (Chapter 3). It achieves the same goal as RLHF, aligning a model to human preferences, but without a separate reward model or a reinforcement learning loop. Instead, it trains the model directly on preference data with a single, stable objective: make preferred answers more likely and rejected ones less likely, while staying close to the original model.

Its guiding insight is that "your language model is secretly a reward model," meaning the reward step can be folded mathematically into the model's own training. DPO is simpler, cheaper, and more stable than full RLHF, which is why it became so widely used.


From the Foundational Modelling glossary

Where DPO is explained

  1. Alignment, Turning a Text Predictor Into a Helpful AssistantFoundational Modelling · Paper 2: DPO, the same goal with far less machinery
  2. Start Here: Foundational Modelling, Explained SimplyFoundational Modelling · The big picture
  3. Scaling Laws and Chinchilla, How Big Should a Model Be?Foundational Modelling · The one-sentence takeaway

Every term, A to Z