Red Teaming
Proactively simulating attackers to find security weaknesses in a model or AI system before release.
In plain language
AI red teaming is not just testing whether a model says bad things; it covers prompt injection, tool privilege escalation, data leakage, bias, hallucination and misuse. Effective programs maintain an attack library, rank by risk, record reproducible inputs and system versions, and separate model issues from application issues. Findings must feed release gates and monitoring, not just a report. High-risk systems should use external red teams and realistic abuse exercises.
Example
Red-team a support agent by inducing it to query another user’s order, paste internal pricing or issue refunds based on a malicious email.
Recommended tools
Related terms
Guardrails
Rules and systems that constrain model inputs, outputs and actions.
Jailbreak
An attack that bypasses model safety limits through role-play, encoding or multi-turn manipulation.
Prompt Injection
An attack that hides malicious instructions in user input or external content to hijack model behavior.
Agent Memory
A mechanism for an agent to store and retrieve user preferences, past tasks and key facts across sessions.