Reinforcement Learning from Human Feedback
Train a reward model from human preferences, then optimize model outputs with reinforcement learning.
In plain language
RLHF is key to making models helpful and aligned with human preferences, often used for chat models. It involves human rankings, reward modeling and reinforcement learning. It is expensive and can introduce reward hacking and preference bias.
Example
Annotators rank two answers, and the reward model learns to prefer helpful, safe responses.
Recommended tools
Related terms
Direct Preference Optimization
An alignment method that optimizes directly on preference data without a separate reward model.
Large Language Model
A model trained on massive text to understand and generate natural language.
Model Evaluation
Designing test sets and scoring rules to decide whether a model works for a specific task.
Agentic Workflow
Breaking a task into steps planned by AI, executed by tools and reviewed by humans.