"LLM-as-a-judge" is the evaluation pattern where, instead of paying humans to grade thousands of model responses, you have a second language model do the grading against a rubric. Amazon Bedrock offers this as a managed job type: you pick a generator model (the one being tested), an evaluator model (the judge), a prompt dataset, and a set of metrics. Bedrock runs the generator over your prompts, has the judge score each response with a written explanation, and gives you score distributions in the console plus a full report in your S3 bucket.
How a judge-model job is put together
Per AWS's documentation, a judge-model evaluation job needs two model choices. The generator side is broad: any Bedrock foundation model (which includes the Claude family), Marketplace models, customized or imported foundation models, prompt routers, and Provisioned Throughput models. Models invoked through the OpenAI Responses API on the bedrock-mantle endpoint can also be generators, though only via the CLI or API, not the console. You can even skip the generator entirely and bring pre-generated responses — covered in the bring-your-own-inference article.
On the metrics side, Bedrock offers selectable built-in metrics — each one drives the judge with a different evaluator prompt — plus user-defined custom metrics where you write the rubric yourself. Cross-Region inference profiles are supported for the listed judge models, so judge capacity can be served across regions.
Which Claude models can be the judge
This is where teams get tripped up. Bedrock's documented evaluator list includes many Claude models — but it tops out at the 4.5 generation. As of July 2026, AWS documents these Claude judges for built-in metrics:
| Documented judge model | Notes |
|---|---|
| Claude Opus 4.5 | Legacy date-versioned ID: anthropic.claude-opus-4-5-20251101-v1:0 |
| Claude Sonnet 4.5 | Newest Sonnet on the judge list |
| Claude Haiku 4.5 | anthropic.claude-haiku-4-5-20251001-v1:0 |
| Claude Sonnet 4, 3.7 Sonnet, 3.5 Sonnet v1/v2 | Older generations |
| Claude 3.5 Haiku, Claude 3 Haiku | Older, cheaper judges |
Non-Claude judges (Amazon Nova, Meta Llama 3.1 70B, Mistral Large) are also listed, and a largely overlapping Claude set — again through Opus 4.5 / Sonnet 4.5 / Haiku 4.5 — is documented for custom metrics.
Pairing generator and judge sensibly
The asymmetry — newest Claude as generator, older Claude as judge — is less of a problem than it looks. Anthropic's own evaluation guidance notes it is "generally best practice to use a different model to evaluate than the model used to generate the evaluated output." A deliberately different judge reduces the risk of a model systematically favoring its own style. A practical default: generate with the model you actually plan to ship (say, Claude Opus 4.8 or Sonnet 5), and judge with Claude Opus 4.5 for nuanced rubrics or Claude Haiku 4.5 when you're scoring high volumes and want to keep judge costs down. The reasoning behind judge independence is covered in its own article.
Whichever judge you pick, invest in the rubric. Bedrock's custom metrics let you define exactly what "good" means; vague criteria produce noisy scores no matter which model grades. Anthropic's guidance for LLM graders — detailed rubrics, forced empirical outputs, letting the judge reason before scoring — applies directly to the evaluator prompts you write here.
What you get back
For each response, the judge produces a score and an explanation of that score. The Bedrock console shows the scores and a histogram of their distribution across the dataset; the complete report, including per-response detail, lands in the S3 bucket you specify when creating the job. That per-response explanation is the underrated part — when a prompt version scores worse, the judge's explanations usually tell you why without manually re-reading every output.
Judge jobs are one of three Bedrock model-evaluation types; the others are programmatic (automatic) jobs with algorithmic metrics and human-worker jobs. If your task has exact expected answers, start with automatic jobs — they're cheaper and more deterministic — and reserve the judge for qualities only another model can assess.
Where to go next
See bring-your-own-inference evaluation for judging Claude outputs generated outside Bedrock, and Knowledge Base evaluation for the RAG-specific variant. The platform overview covers Bedrock's place in the broader Claude 3P picture.