Every model launch comes with a table of benchmark scores, and every enterprise evaluation deck ends up quoting them. The problem is that public benchmarks measure performance on their tasks — academic reasoning tests, competition math, open-source coding challenges — not on yours. Your task has its own documents, vocabulary, edge cases, and definition of "good." A model that leads a leaderboard can still underperform on your workload, and a cheaper model can quietly be more than good enough. An evaluation, or "eval," is simply a repeatable test: a fixed set of real inputs, an expected outcome for each, and a way to score the model's answers.
What Anthropic's own guidance says
Anthropic's prompt engineering documentation is explicit about the order of operations: before you tune a single word of a prompt, you need a clear definition of success criteria, empirical tests against those criteria, and a first-draft prompt to improve. Prompt engineering without an eval is guesswork — you cannot tell whether a change helped, hurt, or did nothing.
The same guidance makes a point that saves teams a lot of wasted effort: not every failing eval is a prompting problem. If your issue is latency or cost rather than accuracy, switching to a different model is often the easier fix than rewriting prompts. Your own eval is what tells you which situation you are in.
Building a minimal eval
You do not need an ML team or a platform purchase to start. A useful first eval is a spreadsheet's worth of work:
Collect 50–200 real examples. Pull actual tickets, contracts, emails, or transcripts from the workflow you want to automate — including the messy ones. Synthetic examples written by the same person who wrote the prompt tend to be suspiciously easy.
Define pass/fail per example. For extraction and classification tasks, this is exact-match scoring a script can do. For summaries or drafts, write a short rubric and have a reviewer (human, or a second model call) grade against it.
Run every candidate through the same set. Same inputs, same scoring, for each model or prompt variant you are comparing. Now the comparison means something.
Keeping eval runs cheap
Evals are repetitive by design, which makes them a natural fit for batch processing. On the Claude API and Claude Platform on AWS, the Message Batches API charges 50% of standard prices, and most batches finish in under an hour — good economics for re-running a few hundred test cases after every prompt change. Amazon Bedrock and Google Vertex AI do not offer Anthropic's Message Batches API, but each has its own cloud-native batch inference for Claude (S3-based on Bedrock; BigQuery/GCS-based on Vertex AI) that serves the same purpose.
from anthropic import AnthropicVertex
client = AnthropicVertex(project_id="my-project", region="global")
scores = []
for case in test_cases: # your 50-200 real examples
msg = client.messages.create(
model="claude-sonnet-5",
max_tokens=1024,
system=CANDIDATE_PROMPT,
messages=[{"role": "user", "content": case["input"]}],
)
scores.append(grade(msg.content[0].text, case["expected"]))
print(f"pass rate: {sum(scores) / len(scores):.0%}")If your prompt shares a long common prefix across every test case (a system prompt, a glossary, reference documents), prompt caching cuts the input cost of the repeated portion to one-tenth of the base rate on cache hits — another reason eval runs cost less than teams expect.
Why this matters more on 3P
On third-party platforms, model choice is also a procurement and quota decision: which models your cloud has enabled, what regional availability looks like, and how batch options differ per platform. A benchmark table cannot answer "will Sonnet 5 at a third of Opus 4.8's input price hit our accuracy bar?" — a one-day eval run can, and the answer directly changes your cost model. Teams that skip this step routinely pay for a top-tier model on workloads a mid-tier model passes comfortably.
Benchmarks are fine as a shortlist filter. They tell you which models are worth testing. Your eval tells you which one to ship.
Where to go next
See building a prompt evaluation framework for a deeper treatment, assessing prompt changes before deployment for the ongoing discipline, and model tiering for turning eval results into cost savings.