Direct Preference Optimization
An alignment method that optimizes directly on preference data without a separate reward model.
In plain language
DPO is simpler and more stable than RLHF, with lower training cost, suiting preference alignment and style tuning. It still depends on high-quality preference data and is sensitive to hyperparameters. Many open models use DPO or similar methods in post-training.
Example
Optimize directly on 10k chosen/rejected answer pairs.
Recommended tools
Related terms
Fine-tuning
Continue training a model on task data to better match a style or task.
Model Evaluation
Designing test sets and scoring rules to decide whether a model works for a specific task.
Reinforcement Learning from Human Feedback
Train a reward model from human preferences, then optimize model outputs with reinforcement learning.
Agentic Workflow
Breaking a task into steps planned by AI, executed by tools and reviewed by humans.