Interpretability
Research into how models internally represent features and make decisions.
In plain language
Interpretability has two layers: behavioral (how input changes affect output) and mechanistic (how neurons, circuits and features combine). Methods include attention visualization, probes, causal interventions, sparse autoencoders and circuit analysis. It matters for safety auditing, debugging and compliance, but remains incomplete: attention weights are not causal importance, and model-generated explanations may be post-hoc rationalization. Treat interpretability as one piece of evidence, not a substitute for external evaluation.
Example
If a feature corresponds to “legal disclaimer,” intervene on it and observe output changes instead of relying on attention heatmaps.
Recommended tools
Related terms
Alignment
Research and engineering to make AI systems behave according to human intent, values and safety constraints.
Benchmark
A standardized dataset and metric for comparing model capabilities.
Hallucination
When a model produces plausible but factually wrong or unsupported content.
Agent Memory
A mechanism for an agent to store and retrieve user preferences, past tasks and key facts across sessions.