Red-teaming an LLM application means deliberately trying to make your own system misbehave — leak instructions, act outside its remit, produce content your policies forbid — before real users or real adversaries do. The name comes from military and security practice, but the enterprise version is defensive quality assurance: you are testing your controls, on your application, with your organization's permission. Nothing in this article is about attacking anyone else's systems.
Why evals aren't enough
Standard evaluations measure how well the system does what it should. Red-teaming probes whether it can be made to do what it shouldn't — a different question with a different mindset. An eval suite checks that the support bot answers billing questions accurately; a red team checks whether a crafted "billing question" can make it reveal its system prompt, discuss another customer's account, or promise a refund policy that doesn't exist. Model-side safeguards help — newer Claude models can refuse with an explicit stop_reason: "refusal", and on Claude Fable 5 safety-classifier refusals include a stop_details.category — but those protect against broad harm categories, not against your application-specific failure modes. Only you know that "quote a discount above 20%" is a violation in your business.
Why it belongs in pre-launch review
Findings are cheapest before launch: no incident disclosure, no customer harm, no production freeze while you patch. A short adversarial exercise fits naturally beside the security review in your approval gates, and its output — a set of concrete prompts that break things — becomes a permanent regression suite. Post-launch, repeat the exercise when the model, the system prompt, or the tool set changes materially, since each of those shifts the attack surface.
Scoping a first exercise
Keep the first one small and concrete:
- One application, written goals. Pick a single app near launch and state what "misbehavior" means for it in five or six testable claims ("cannot be induced to reveal its system prompt," "will not discuss competitor pricing," "never executes a tool call on unverified user identity").
- Isolated environment. Run against a staging deployment with no production customer data. On the Claude API side, a dedicated workspace with its own API keys and a workspace spend limit keeps the exercise contained and separately metered; archiving the workspace afterward immediately revokes all its API keys.
- A few days, a few people. Two to four testers for two to five days finds most of the obvious problems. Include one person who did not build the system.
- Log everything. Every attempt, verbatim, with the model ID and prompt version — findings you can't reproduce are anecdotes.
One provider-side note: adversarial content can resemble genuine misuse, and Anthropic's documented trust-and-safety exception allows content flagged as a Usage Policy violation to be retained for up to two years (classification scores up to seven). Defensive testing of your own application is a normal practice, but stay within your provider's acceptable-use terms, keep the exercise clearly documented, and raise questions with your account representative if scope is unclear.
What to do with findings
A finding is a reproducible prompt plus the observed bad behavior. Triage each by severity and fix at the right layer: input validation and context isolation for injection paths, system prompt hardening for instruction-following failures, output filtering and human gates for consequential actions. Then close the loop twice — re-test the specific finding, and add it to your permanent eval suite with the results recorded per eval documentation practice, so the regression can never return silently. Findings that indicate active abuse potential should also feed your misuse monitoring signals.
Where to go next
When you're ready to formalize, scoping a red team exercise covers roles and rules of engagement, and the adversarial prompt catalogue lists the attack families to test. For the underlying attack concept, start with prompt injection 101.