Anthropic's testing documentation opens with a claim worth taking literally: "Building a successful LLM-based application starts with clearly defining your success criteria and then designing evaluations to measure performance against them." Not with picking a model, not with writing prompts — with defining success. The framework the docs use for that definition is a familiar one from management training, applied to AI: criteria should be Specific, Measurable, Achievable, and Relevant.
The four letters, applied to Claude
Specific. "Good performance" is not a criterion; it's a hope. The docs' example of the fix: instead of "good performance," specify "accurate sentiment classification." Naming the exact task forces you to name the exact failure modes you care about, which is what your evaluation will actually test.
Measurable. Every criterion needs a number attached — a score, a percentage, a count, a time. Anthropic's docs list concrete options: task-specific metrics such as F1 score, BLEU, or perplexity; generic metrics such as accuracy, precision, and recall; and operational metrics such as response time in milliseconds or uptime percentage. Qualitative judgments can be made measurable too, via Likert scales (numbered rating scales, typically 1–5), expert rubrics, A/B testing, user feedback, and edge-case analysis.
Achievable. The target must be grounded in something — industry benchmarks, prior experiments, published AI research, or expert knowledge. The docs are blunt: "Your success metrics should not be unrealistic to current frontier model capabilities." Demanding 100% accuracy on an ambiguous task guarantees your project fails on paper even when it succeeds in practice.
Relevant. The metric should track what your business actually needs. A support bot optimized purely for factual accuracy but never measured on tone may pass every eval and still generate complaints.
"But our goal is fuzzy" — quantify it anyway
The most common objection is that some goals resist numbers. Anthropic's docs push back with a worked contrast. Bad criterion: "Safe outputs." Good criterion: "Less than 0.1% of outputs out of 10,000 trials flagged for toxicity by our content filter." The second version names the sample size, the threshold, and the measuring instrument. Anyone on the team can rerun it in six months and get a comparable answer — which is exactly what you need when you migrate to a new Claude model and want to know whether behavior shifted.
The worked example: an F1 target dissected
Anthropic's docs give a model answer for a sentiment-analysis use case: "an F1 score of at least 0.85 ... on a held-out test set of 10,000 diverse Twitter posts ... a 5% improvement over our current baseline." F1 score, if the term is new, is a standard classification metric that balances precision (of the items you labeled positive, how many really were?) against recall (of the truly positive items, how many did you catch?). It punishes both over-flagging and under-flagging, which plain accuracy does not.
Notice what each clause is doing:
| Clause | What it pins down |
|---|---|
| "F1 score of at least 0.85" | The metric and the threshold — Specific and Measurable |
| "held-out test set" | Data the prompt was never tuned on, so the score isn't flattered by memorization |
| "10,000 diverse Twitter posts" | Sample size and distribution — large enough that the number is stable, drawn from the real input domain |
| "5% improvement over our current baseline" | Achievable and Relevant — anchored to something you've already measured |
You can transplant this skeleton onto almost any task: [metric] of at least [threshold] on a held-out set of [N] [realistic inputs], beating [baseline] by [margin].
One criterion is rarely enough
Anthropic's docs list the dimensions most applications end up caring about: task fidelity, consistency, relevance and coherence, tone and style, privacy preservation, context utilization, latency, and price — and note that "most use cases will need multidimensional evaluation along several success criteria." A criterion per dimension keeps trade-offs visible instead of hidden inside one blended score. How to weight those dimensions when they conflict is its own topic — see multidimensional evaluation.
Once your criteria exist, the next step is building graders that check them automatically: code-based grading for anything with a deterministic answer, and LLM-as-judge rubrics for the rest. If you have no test data at all yet, the Console test case generator can bootstrap a starter set.
Where to go next
This applies identically whether you call Claude first-party or through Bedrock, Vertex AI, or Foundry — success criteria live in your process, not in any platform. For platform-specific eval tooling, start from the articles index or the feature matrix.