Anthropic's evaluation docs grade the three grading methods with striking candor. Code-based grading: fastest, most reliable, extremely scalable. LLM-based grading: fast, flexible, scalable. Human grading: "Most flexible and high quality, but slow and expensive. Avoid if possible." The companion principle is just as direct — "Prioritize volume over quality: More questions with slightly lower signal automated grading is better than fewer questions with high-quality human hand-graded evals." A 2,000-case automated suite you run on every prompt change beats a 50-case hand-graded set you run twice a year, because evaluation value compounds with repetition and humans don't repeat cheaply.
Where human grading is genuinely unavoidable
"Avoid if possible" implies it isn't always possible. A few situations resist automation:
Calibrating the automated graders themselves. Before you trust an LLM judge, Anthropic's docs say to test its reliability first. The only reference standard available is human judgment: grade a sample by hand, run the judge on the same sample, and measure agreement. This is the highest-leverage human grading you'll ever do, because a few hundred careful labels validate a grader that then scores everything else for free.
Genuinely ambiguous cases. Anthropic's list of edge cases to include in evals ends with "ambiguous test cases where even humans would find it hard to reach an assessment consensus." An LLM judge scoring these produces confident noise; what you actually need to know is how humans split, which only humans can tell you.
Judgments that require accountable expertise. Whether a clinical summary is safe, a contract clause is acceptable, or a response meets a regulator's expectations are questions where organizations need a qualified person's verdict on record, not a model's. Amazon Bedrock's evaluation service acknowledges this category explicitly: alongside programmatic and judge-model jobs, it offers model evaluation jobs that use human workers — "employees of your company or a group of subject-matter experts" — as the graders.
Taste and preference at launch. Early in a product's life, before you know what "good" means precisely enough to write a rubric, human review of real outputs is how the rubric gets discovered. The output of that phase should be a written rubric — which is precisely what lets you stop doing it.
How to keep labeling sessions small and consistent
When humans must grade, the enemy is inconsistency: two annotators reading the same output and reaching different verdicts. Every technique that makes LLM judges reliable was borrowed from human annotation practice, and they all apply in the original direction:
Write the rubric down. Detailed criteria with explicit auto-fail rules ("fail if the required disclaimer is missing") leave less room for interpretation than "rate the quality."
Force a constrained verdict. Binary pass/fail or a defined 1–5 Likert scale with a sentence describing each point. Free-text feedback is valuable for discovering failure modes but cannot be aggregated into a score.
Overlap a slice. Have every annotator grade the same subset so you can see agreement directly. If they disagree often, the rubric — not the annotators — needs fixing. Disagreement cases feed the "ambiguous" bucket of your edge-case taxonomy.
Grade one dimension at a time. Asking someone to simultaneously judge accuracy, tone, and completeness in one pass produces blended, drifting scores. Separate passes per dimension are faster per judgment and far more consistent.
Shrinking the human share over time
The sustainable pattern is a funnel. Automated graders — code-based checks first, LLM judges second — score everything on every run. Humans see only what automation can't settle: judge-versus-human calibration samples, flagged ambiguous cases, and periodic spot-checks to confirm the judge hasn't drifted. Anthropic's docs also suggest letting Claude help brainstorm evaluation methods and generate test cases from a baseline set, which shrinks the human authoring burden as well as the grading one — see bootstrapping an eval set.
Where to go next
Start from SMART success criteria so your rubric has something concrete to encode, and read volume over quality for the strategic case behind Anthropic's ranking of grading methods.