Incident response for an LLM application uses the same discipline as any other production incident: detect, contain, assess, remediate, learn. What changes is the catalog of likely incidents. You are less likely to face a novel exploit and more likely to face a screenshot of your assistant saying something wrong, a developer realizing a prompt contained data it should not have, or a launch-day traffic spike exhausting your quota. This article gives you a starter runbook for each, plus the preparation that makes all three manageable.
Before anything happens: three preparations
First, a kill switch. Every LLM feature needs a way to be disabled quickly without redeploying: a feature flag that swaps the AI path for a graceful fallback ("this feature is temporarily unavailable") beats an emergency code change at 2 a.m. If your traffic flows through an internal gateway, that is the natural place for it.
Second, reconstructable history. You cannot assess an incident you cannot replay. Decide in advance what you log (request metadata always; content according to your privacy policy) and confirm you can answer "what did the model actually say to this user last Tuesday?" within your retention window.
Third, named ownership. One person owns each LLM feature's incident path, and the general on-call knows these runbooks exist. Ambiguity about whether an AI incident belongs to security, engineering, or "the AI team" wastes the first hour.
Runbook 1: Bad output reached users
The model produced something wrong, offensive, or off-policy, and a user saw it. Contain: judge blast radius first. One odd response to one user is feedback; a repeating pattern, or bad output in an automated pipeline that acts without review, justifies the kill switch. Assess: pull the logged conversation and determine why: a prompt regression from a recent change, adversarial user input, or ordinary model error on a hard case. Check whether the same trigger affects other users. Remediate: fix the proximate cause (revert the prompt change, add an output check, tighten instructions), and where the output caused real harm, treat user communication as part of remediation, not PR. Learn: add the failing case to your evaluation set so the regression cannot return silently. Over time, this loop is what turns incidents into test coverage.
Runbook 2: Data went where it shouldn't
This family includes: sensitive data included in prompts against policy, prompt or response content exposed through your own logs, or a system prompt revealed to users. Contain: stop the flow first (disable the offending pipeline or field), then secure your own stores, since your logs are usually the largest copy of the exposed data; restrict access before the investigation grows the audience. Assess: what data, whose, over what period, and to which systems? For data sent to the platform itself, your provider's data-handling and retention terms determine what exists on their side; involve your privacy team early, as regulatory notification clocks may apply depending on the data and jurisdiction, and confirm specifics with your provider rather than assuming. A leaked system prompt deserves proportionate response: it is rarely an emergency by itself, but treat any credentials or sensitive details embedded in it as compromised, and note that a well-designed system prompt should never contain secrets in the first place. Remediate and learn: the durable fixes are structural: minimization and redaction before requests are sent, logging policies that store less, and tests that fail the build when known sensitive patterns pass through unredacted.
Runbook 3: Quota exhaustion or platform outage
The unglamorous incident that will probably happen first. Symptoms: rate-limit errors, rising latency, or a regional outage. Contain: your app should already degrade gracefully: retries with exponential backoff for transient errors, queuing for deferrable work, and honest messaging for interactive features. If one feature is starving others, shed the low-priority load first; this is much easier if traffic is centrally routed. Assess: distinguish "we outgrew our quota" (a forecasting miss) from "the platform is degraded" (check your provider's status channels). Remediate: request quota increases through your cloud provider before launch days, not during them; consider routing less demanding requests to a cheaper, faster model like Claude Haiku 4.5 under pressure; and if availability requirements are strict, evaluate whether a second platform is a justified fallback, remembering that feature availability differs between platforms, so a fallback must be tested against the features you actually use.
Where to go next
The detection side of this article lives in Observability for Claude Apps, and the graceful-degradation patterns in Retries, Timeouts, and Fallbacks. To see how incident planning fits the broader review, read The Security Review: What Your CISO Will Ask.