Validation is the gate between "works in the demo" and "users depend on it." Done badly, it is a rubber stamp or an endless committee loop. Done well, it is a checklist of questions the building team answers with evidence, reviewed by people with the standing to say no. What follows is a recommended structure for that review, scaled by risk tier — a low-risk internal summarizer might need a lightweight pass; a customer-facing system deserves the full treatment. None of this substitutes for obligations your regulator or contracts impose.
The questions to answer, in five groups
Fitness for purpose. What exactly does the system do, for whom, and what happens when it is wrong? Evidence: a written use-case description with explicit boundaries ("answers questions about our published docs; does not give pricing commitments") and results from an evaluation set representative of real inputs, including hard and adversarial cases. The eval set, the scores, and the pass thresholds should be recorded as evaluation documentation — they become your regression baseline later.
Configuration identity. Precisely what is being launched? Evidence: the exact model ID, platform, system prompt version, tool definitions, and retrieval sources. Model IDs matter more than they look: every Claude model ID is a pinned snapshot (dateless IDs like claude-opus-4-8 included), so recording the exact ID makes your validation reproducible — and model behavior genuinely differs across versions. For example, Anthropic's migration guide notes that moving from Sonnet 4.6 to Sonnet 5 changes the tokenizer (roughly 30% more tokens for the same text) and rejects previously valid parameters. Validation evidence collected on one model does not automatically transfer to another.
Data handling. What data flows in and out, at what sensitivity, and under what terms? Evidence: a data-flow sketch and confirmation against your platform's documented posture — for the Claude API, Anthropic states inputs and outputs are deleted within 30 days by default and are not used for training without express permission; on Bedrock and Google Cloud, the cloud provider is the data processor and its policies govern. If your organization has a zero-data-retention or HIPAA-ready arrangement, verify the feature set you plan to use is actually eligible under it — some stateful features are documented as outside ZDR coverage, and certain models (Claude Fable 5 and Claude Mythos 5) are documented as requiring 30-day retention and are not available under ZDR at all. Catching that at validation is cheap; catching it as a production 400 error is not.
Failure and misuse behavior. What happens on bad input, prompt injection attempts, API errors, or refusals? Evidence: red-team results for the tier, error-handling tests, and — where the tier warrants — a defined human oversight point before consequential actions. Note that newer models surface safety refusals explicitly (a refusal stop reason); your application should handle that path gracefully, and the test proving it belongs in the evidence pack.
Operational readiness. Who owns it after launch? Evidence: a named owner in the inventory, monitoring in place, a rollback plan, and an entry in the incident playbook's scope.
Who signs off
| Stakeholder | Signs off on |
|---|---|
| Use-case owner | Fitness for purpose; accepts residual risk in their domain |
| Engineering lead | Configuration identity, failure handling, operational readiness |
| Security / privacy reviewer | Data handling and access controls |
| Governance program owner | Process completeness; tier-appropriate evidence exists |
| Steering committee | High-tier launches only |
Keep signatures on the evidence pack itself, dated, in your document system — this is the audit trail.
Revalidation is part of validation
Record, at sign-off, what would trigger a re-review: a model version change (including vendor deprecations — deprecated models have published retirement dates), a material prompt or retrieval change, an expansion of audience or autonomy, or a periodic clock (annually is common for high tiers). That trigger list is the handoff to change management.
Where to go next
See model risk management basics for why these pillars exist, and evaluation and testing for building the eval sets this process depends on.