Most enterprise software testing answers a binary question: does the code do what the spec says? LLM features break that model. A summarization endpoint can return grammatical, confident, plausible text that is subtly wrong — and no unit test will catch it. The discipline that fills this gap is evaluation: running your feature against a fixed set of representative inputs and scoring the outputs, so that quality becomes a number you track instead of an impression you argue about.
What an eval set is
An evaluation set (eval set) is a collection of real or realistic inputs paired with a definition of what a good output looks like. For a contract-summary feature, that might be fifty contracts with reference summaries written by your legal team. For a ticket classifier, five hundred tickets with correct categories. For a support assistant, a hundred customer questions with notes on what a correct, on-policy answer must contain.
The set does not need to be large to be useful. A few dozen carefully chosen cases will catch most regressions; a few hundred gives you statistically meaningful comparisons. What matters more than size is coverage: include the common cases, the known-hard cases, and the cases where a wrong answer would be expensive — the ones your risk review flagged.
Three ways to score outputs
How you grade depends on the task, and most programs use a mix.
| Method | Works for | Trade-off |
|---|---|---|
| Exact / programmatic checks | Classification, extraction, structured outputs | Cheap and objective, but only fits tasks with checkable answers |
| Model-graded review | Summaries, drafts, open-ended answers | Scales well; the grading prompt itself needs spot-checking by humans |
| Human review | High-stakes or nuanced quality judgments | The gold standard, but slow and costly — reserve it for samples and audits |
Programmatic checks should be your default wherever the task allows: did the JSON parse, is the category one of the allowed values, does the extracted total match the invoice? Model-graded evaluation — using Claude itself to score outputs against a rubric — handles the fuzzier middle ground, provided you periodically verify the grader agrees with your human experts.
When to run evals
Evaluation is not a launch gate you pass once. Run your eval set at four moments: before launch, to establish a baseline and decide whether the feature is good enough to ship; on every prompt change, because prompts are configuration and one edit can shift behavior broadly; on every model version change, since an upgrade that helps on average can still regress your specific task (see upgrading models without breaking production); and periodically in production, by sampling live traffic and scoring it, because real inputs drift away from your test set over time.
Set thresholds before you measure
Decide in advance what "good enough" means, with the business owner in the room. Is 95% classification accuracy acceptable for routing internal tickets? Probably. For flagging regulatory filings? Probably not — that use case may need a human review step regardless of the score, which is a design question covered in human-in-the-loop design. Writing the threshold down first prevents the natural temptation to declare whatever number you measured to be acceptable.
Also track more than one number. Accuracy alone hides failure patterns: a summarizer can score well on average while consistently failing on scanned documents, or on one language, or on inputs over a certain length. Slice your results by input type so you know where the feature is weak, not just how it performs overall.
Keep it boring and owned
The teams that succeed at this treat evals like any other engineering asset: the eval set lives in version control, results are logged over time, and a named owner curates new cases — especially real production failures, which are the most valuable additions. None of this requires ML researchers. It requires the same habits your organization already applies to regression testing, pointed at a new kind of output.
Where to go next
Evaluation and prompting improve together — read prompt engineering for business applications next, and human-in-the-loop design for what to do when scores alone aren't enough. The launch checklist shows where evals fit in a rollout.