LLM-as-judge — sometimes called LLM-based grading — means sending a model's output to a second model along with grading instructions, and letting the second model assign the score. Anthropic's docs summarize the profile: "Fast and flexible, scalable and suitable for complex judgement. Test to ensure reliability first then scale." That last sentence is the catch. A judge is only as reliable as its rubric, and the docs give three specific techniques for making rubrics that produce stable, usable scores.
Technique 1: detailed rubrics with auto-fail rules
A rubric is the written standard the judge grades against. Vague rubrics ("rate the quality of this answer") force the judge to invent criteria on the fly, differently for each test case. Anthropic's guidance is to make the rubric detailed and clear — and its documented example is an auto-fail rule: fail the answer automatically if it doesn't mention "Acme Inc." in the first sentence. Auto-fail rules are powerful because they convert your most important requirements into binary checks the judge can't fudge. Legal disclaimers, required citations, forbidden content, mandatory structure — anything non-negotiable belongs in an explicit "score 0 if..." clause rather than in a vague plea for quality. Complex use cases, the docs note, "might require several rubrics for holistic evaluation" — one for accuracy, one for tone, one for policy compliance — rather than one rubric trying to weigh everything at once.
Technique 2: force an empirical output
Left to itself, a judge model will write a paragraph of nuanced commentary that no script can aggregate. Anthropic's docs say to force the output into a fixed vocabulary: "output only 'correct' or 'incorrect'," or a numeric scale of 1–5 (a Likert scale). Binary outputs are the most reliable and easiest to threshold; Likert scales preserve gradations when you need them, at some cost in judge consistency. Either way, your harness should parse the verdict from a predictable location — a dedicated XML tag works well — and treat unparseable output as a grading error, not silently as a pass or fail.
Technique 3: encourage reasoning, then discard it
The most counterintuitive documented tip: "Ask the LLM to think first before deciding an evaluation score, and then discard the reasoning." Having the judge reason step-by-step before answering improves the quality of the verdict — but the reasoning text itself is scaffolding, not data. Your score is the token in the answer tag; the chain-of-thought is thrown away rather than being averaged, mined, or shown as justification. Keeping the two separated prevents the failure mode where a fluent-sounding explanation makes a wrong verdict look trustworthy.
from anthropic import AnthropicVertex
client = AnthropicVertex(project_id="my-project", region="global")
rubric = """Grade the answer against this rubric:
- Auto-fail (score: incorrect) if 'Acme Inc.' is missing from sentence 1.
- Otherwise: correct only if the refund policy is stated accurately.
Think step by step in <thinking> tags, then output exactly
'correct' or 'incorrect' in <verdict> tags."""
msg = client.messages.create(
model="claude-opus-4-8", max_tokens=1024,
messages=[{"role": "user",
"content": f"{rubric}\n\nAnswer to grade:\n{output}"}])
verdict = msg.content[0].text.split("<verdict>")[1].split("</verdict>")[0]The same discipline on managed platforms
The rubric techniques transfer directly to the clouds' managed judge tooling. Amazon Bedrock's LLM-as-a-judge evaluation jobs pair a generator model with an evaluator model and support both built-in metrics (each with its own evaluator prompt) and user-defined custom metrics — your custom metric prompt is exactly the kind of rubric described above, and Claude models through Opus 4.5, Sonnet 4.5, and Haiku 4.5 are documented evaluator options there. Microsoft Foundry's LLM-judge evaluators score on a 1–5 Likert scale with a default pass threshold of 3 — the forced-scale idea productized — though Foundry only documents Azure OpenAI and OpenAI models as judges, not Claude. Vertex AI's Gen AI evaluation service supports model-based pointwise and pairwise metrics with a configurable judge model. Which model should do the judging — and why it should differ from the model being judged — is covered in judge model independence.
Where to go next
Ground your rubric in explicit success criteria, use code-based grading for everything deterministic first, and see the Bedrock judge setup guide for the managed-job version of this workflow.