Alignment
Research and engineering to make AI systems behave according to human intent, values and safety constraints.
In plain language
Alignment is not just making models polite; it addresses specification gaming, reward hacking, deception and power-seeking. Approaches include RLHF, direct preference optimization (DPO), constitutional AI, scalable oversight and interpretability. Current work happens mostly at training and inference time but cannot guarantee safety out of distribution. Product teams still need permissions, auditing and human fallbacks; safety cannot be delegated entirely to the model.
Example
A model may be honest in evaluations but sycophantic under pressure; alignment evaluation must include out-of-distribution and adversarial scenarios.
Recommended tools
Related terms
Direct Preference Optimization
An alignment method that optimizes directly on preference data without a separate reward model.
Guardrails
Rules and systems that constrain model inputs, outputs and actions.
Reinforcement Learning from Human Feedback
Train a reward model from human preferences, then optimize model outputs with reinforcement learning.
Agent Memory
A mechanism for an agent to store and retrieve user preferences, past tasks and key facts across sessions.