Code-Based Grading: Exact Match, String Match, and When Each Applies

Anthropic's evaluation docs rank grading methods by a simple rule: "choose the fastest, most reliable, most scalable method" that your task allows. At the top of that ranking sits code-based grading — deterministic checks written in ordinary code. The docs describe it as "fastest and most reliable, extremely scalable, but also lacks nuance for more complex judgements." The two workhorses are exact match and string match.

The two graders, in Python

Exact match asks whether the output equals the expected answer, character for character: output == golden_answer. String match asks whether a required phrase appears anywhere in the output: key_phrase in output. Both come straight from Anthropic's documented examples, and both fit in a grading harness like this:

from anthropic import AnthropicBedrockMantle

client = AnthropicBedrockMantle(aws_region="us-east-1")

def grade(test_case):
    reply = client.messages.create(
        model="anthropic.claude-sonnet-5",
        max_tokens=64,
        messages=[{"role": "user", "content": test_case["input"]}],
    )
    output = reply.content[0].text.strip()
    exact = output == test_case["golden_answer"]      # exact match
    contains = test_case["key_phrase"] in output       # string match
    return exact, contains

Run that over a labeled dataset and you get accuracy numbers in minutes, at no grading cost beyond the generation itself. Anthropic's own example recipe uses exact match to score sentiment classification over 1,000 labeled tweets — a scale where human grading would take days and an LLM judge would add cost for no extra signal.

Which tasks each grader suits

A practical pattern is to design the task for the grader: Anthropic's eval-building principles include "Automate when possible: Structure questions to allow for automated grading (e.g., multiple-choice, string match, code-graded, LLM-graded)." If you instruct Claude to answer with exactly one label from a fixed list — or use structured outputs to guarantee a JSON schema — you convert a fuzzy generation problem into an exact-match problem. That structure pays for itself every time the eval runs.

What code-based grading misses

The documented weakness — "lacks nuance" — shows up quickly on open-ended tasks. Exact match can't tell an eloquent, correct summary from a garbled one that happens to reuse the reference wording; string match can't tell whether the required phrase appears in a sentence that says the opposite of what you intended. Between pure string checks and LLM judges sits a middle tier of computed similarity metrics, and Anthropic's docs include recipes for them: ROUGE-L for summarization quality against reference summaries, and cosine similarity over sentence embeddings for checking that paraphrased questions get consistent answers. These are still code — deterministic and cheap — but they score closeness rather than equality.

When the judgment is genuinely qualitative — tone, helpfulness, groundedness — move up to LLM-as-judge grading, and keep human grading as the last resort the docs say it should be. The volume principle matters here too: Anthropic explicitly advises prioritizing more questions with slightly lower-signal automated grading over fewer, hand-graded ones — see volume over quality.

The same idea on the cloud platforms

Code-based grading is portable by nature — the harness above works unchanged against Bedrock, Vertex AI, or Foundry endpoints by swapping the client class. The platforms also offer managed versions of the concept: Amazon Bedrock's automatic evaluation jobs compute deterministic metrics such as accuracy and word-error-rate over built-in or custom datasets; Vertex AI's Gen AI evaluation service lists computation-based metrics including exact match, BLEU, and ROUGE; and Microsoft Foundry's built-in evaluators include F1 score, BLEU, GLEU, ROUGE, and METEOR alongside a String Checker grader. Whether Claude can be the generator under each managed tool varies by platform — Bedrock documents it broadly, while Foundry does not document Claude as an evaluation target — so for maximum portability, many teams simply keep the grader in their own code.

Where to go next

Define thresholds first with SMART success criteria, bootstrap a labeled dataset with the Console test case generator, then wire the harness into CI — see evals in CI.

Sources

• Anthropic — Define success criteria and build evaluations
• Amazon Bedrock — Model evaluation task types and metrics
• Google Vertex AI — Determine your evaluation approach
• Microsoft Foundry — Built-in evaluators