An evaluation ("eval") is a repeatable test of model behavior: a fixed set of inputs, a way to score the outputs, and a threshold that defines "good enough." Most teams building on Claude eventually run evals — before launch, before a prompt change, before a model upgrade. Far fewer keep records that would let them answer, a year later, which configuration passed which tests at what level, and who accepted the result. That record is cheap to keep and expensive to reconstruct.
What to record for every run
| Field | Notes |
|---|---|
| Test set identifier and version | The exact dataset, versioned like code. Note provenance and whether it contains real customer data. |
| Model ID | Claude model IDs are pinned snapshots, so recording the ID (e.g. claude-opus-4-8) fixes the model side of reproducibility. |
| Prompt version and parameters | System prompt version, tool definitions, and settings such as max_tokens — see system prompt version control. |
| Metrics and thresholds | What was measured, the score, and the pre-agreed pass/fail line. A threshold decided after seeing results is not a threshold. |
| Run date, environment, and cost | Which workspace/platform it ran on; token usage for budgeting. |
| Reviewer sign-off | A named person who accepted the result, with date. Ideally not the person who wrote the prompt. |
| Exceptions and known failures | Cases it still fails, and why that was accepted. This is the field auditors care about most. |
Store these as structured entries (one file or row per run) in version control or a database — not as screenshots in a chat thread. Link each production deployment to the eval run that cleared it.
Reproducibility: what holds still and what doesn't
Pinned model snapshots mean a recorded model ID keeps meaning the same model. But models do retire — for example, claude-opus-4-1-20250805 is deprecated and scheduled for retirement on August 5, 2026, with Opus 4.8 as the recommended migration target — so an eval record tied to a retired model documents history rather than something you can re-run. That is fine: the record's job is to show what you knew when you shipped. The corollary is that every migration is a new eval run, not a diff you can skip. Migrations can shift more than quality: the tokenizer used by Opus 4.7 and later, Sonnet 5, and Fable 5 produces roughly 30% more tokens for the same text than earlier models, which moves cost and latency baselines even when accuracy holds.
Make cost per run visible
Run evals under a dedicated workspace or API key. On the Claude API, the Admin Usage API can then report token consumption with breakdowns by model, workspace, and API key, and the cost endpoint reports daily spend — enough to put a dollar figure in each eval record without any custom metering. On Bedrock, Vertex AI, and Foundry, use your cloud provider's native cost attribution the same way.
Answering the audit question
When an internal auditor, customer, or regulator asks how you validated the system, the honest answers come straight out of well-kept run records: what was tested (test set version), against what bar (thresholds), with what result (metrics), accepted by whom (sign-off), and what residual weaknesses were known (exceptions). Frame this as your organization's evidence of diligence — whether it satisfies a specific regulatory requirement is a question for your counsel, and the answer will differ by industry and jurisdiction. What no framework accepts is "we tried some prompts and it seemed fine."
Where to go next
For designing the evals themselves, start with evaluation and testing basics. Adversarial cases discovered in red-team exercises should graduate into your recorded eval suite, and results feed the decision audit trail.