Anthropic's evaluation docs state three design principles, and the third is the one teams resist: "Prioritize volume over quality: More questions with slightly lower signal automated grading is better than fewer questions with high-quality human hand-graded evals." The intuition behind it is statistical. A 20-question hand-graded suite gives you exquisite judgment on 20 data points — too few to detect a real regression from noise, too few to cover edge cases, and too expensive to re-run on every prompt tweak. A 1,000-question automated suite re-runs in minutes for the cost of the tokens, and a grader that is occasionally wrong on individual rows still produces a trustworthy aggregate score.
The economics compound on third-party platforms: whether you run evals against Amazon Bedrock, Vertex AI, or Microsoft Foundry, an eval run is just inference, billed at your normal token rates. Cheap, repeatable runs are what make evaluation a habit instead of a launch-gate ceremony.
The three grading methods — and the pick rule
The docs give a one-line rule for choosing a grader: "choose the fastest, most reliable, most scalable method." They then rank three options.
| Method | Anthropic's verdict | Typical use |
|---|---|---|
| Code-based | "Fastest and most reliable, extremely scalable, but also lacks nuance for more complex judgements" | Exact match, string match, format checks |
| LLM-based | "Fast and flexible, scalable and suitable for complex judgement. Test to ensure reliability first then scale." | Tone, groundedness, rubric scoring |
| Human | "Most flexible and high quality, but slow and expensive. Avoid if possible." | Calibration and spot checks |
Code-based grading is as simple as it sounds — the docs' own examples are an exact match (output == golden_answer) and a string match (key_phrase in output):
def grade_exact(output: str, golden: str) -> bool:
return output.strip() == golden.strip()
def grade_contains(output: str, key_phrase: str) -> bool:
return key_phrase in output
score = sum(grade_exact(o, g) for o, g in results) / len(results)When judgment is genuinely required — is this response empathetic, is it grounded in the provided document — use an LLM grader. The docs' tips: write detailed rubrics (even auto-fail conditions, like requiring 'Acme Inc.' in the first sentence), force empirical outputs ("output only 'correct' or 'incorrect'" or a 1–5 scale), and "Encourage reasoning: Ask the LLM to think first before deciding an evaluation score, and then discard the reasoning." The example grader code also carries a practice worth adopting: use a different model to evaluate than the one that generated the output (more on that in judge model independence).
Getting to volume: Claude writes the test cases
Volume sounds expensive until you stop hand-writing rows. The docs are direct about this: "Writing hundreds of test cases can be hard to do by hand! Get Claude to help you generate more from a baseline set of example test cases." The workflow: author a small seed set yourself — enough examples to pin down the task and its edge cases — then have Claude generate variations, review a sample, and grow the suite. The docs also suggest asking Claude to brainstorm evaluation methods for your criteria, and an evals cookbook with worked code for human-, code-, and LLM-graded evals is linked from the documentation.
The docs' own example recipes show what scale looks like in practice: 1,000 labeled tweets graded by exact match for sentiment, 200 articles scored with ROUGE-L for summarization, 500 simulated queries checked by an LLM for privacy leaks. Note the pattern — hundreds to a thousand rows, all machine-graded.
Where human review still earns its cost
"Avoid if possible" is not "never." Humans earn their cost in three places. First, calibrating the automated graders: before you trust an LLM judge at scale, humans grade a sample and you check agreement — the docs say to "test to ensure reliability first then scale." Second, the ambiguous slice: test cases where even humans disagree cannot be meaningfully auto-graded for correctness. Third, reviewing generated test cases before they enter the suite, so a generation quirk doesn't become a permanent blind spot. In each case the human acts on a sample and the machine handles the volume.
Where to go next
See code-based grading and LLM-judge rubrics for the two automated methods in depth, human grading scope for the sampling strategy, and eval run costs for budgeting runs on your platform.