Amazon Bedrock ships a managed evaluation capability covering "Amazon Bedrock models and knowledge bases, as well as models and Retrieval Augmented Generation (RAG) sources outside of Amazon Bedrock." For model evaluation it offers three job types: programmatic (automatic) jobs, jobs that use human workers — "employees of your company or a group of subject-matter experts" — and jobs that use a judge model, where a second LLM scores each response and explains its score. This article covers the first type: automatic jobs, which compute algorithmic metrics with no LLM judge in the loop, making them the cheapest and most repeatable of the three.
The four task types
Automatic jobs are organized around predefined task types, each with its own metrics and built-in datasets — and a hard structural constraint: one task type per evaluation job. If you want a model scored on both summarization and classification, that is two jobs.
| Task type | Metrics computed | Built-in datasets (examples) |
|---|---|---|
| General text generation | Accuracy (real-world-knowledge / RWK score), Robustness (word error rate), Toxicity | TREX, BOLD, WikiText2, RealToxicityPrompts |
| Text summarization | BERTScore, deltaBERTScore, Toxicity | Gigaword |
| Question and answer | NLP-F1, F1/deltaF1, Toxicity | BoolQ, NaturalQuestions, TriviaQA |
| Text classification | Accuracy, delta-accuracy | Women's Ecommerce Clothing Reviews |
In plain terms: BERTScore and its delta variant measure how semantically close a generated summary is to a reference summary; word error rate measures how much output degrades under perturbed inputs; F1 blends precision and recall for answer overlap; and the toxicity metrics score outputs against datasets built for that purpose, such as RealToxicityPrompts and BOLD.
Built-in versus custom datasets
The built-in datasets are public benchmarks, which makes them useful for like-for-like model comparison and nearly useless for telling you how a model handles your claims-processing emails. Bedrock supports custom prompt datasets for automatic jobs precisely for this reason. The pragmatic split: use built-in datasets when shortlisting models, then build a custom dataset from your own traffic for the decision that matters. A custom dataset also lets you encode the edge cases a public benchmark will never contain.
Which models can you evaluate?
Model evaluation jobs support Bedrock's full range of model sources: foundation models (which is where Claude lives on Bedrock), Marketplace models, customized and imported foundation models, prompt routers, and Provisioned Throughput models. One documentation quirk worth knowing: Bedrock's evaluation pages do not centrally list per-model support — the docs delegate to each model's own page ("To see which models support model evaluation, please visit models at a glance and choose the model you are interested in"). Before planning an evaluation program around a specific Claude model as the generator, confirm on that model's entry.
Bedrock can also evaluate responses you generated elsewhere: with bring-your-own-inference, "Amazon Bedrock skips the model invoke step and directly evaluates the data you supply." That means outputs produced by Claude on any surface — including the first-party API — can be scored by a Bedrock evaluation job (see bring-your-own-inference evaluation).
Where automatic jobs fit in an eval strategy
Automatic jobs implement, in managed form, the "automate when possible" principle from Anthropic's own eval guidance — algorithmic metrics over large question volumes, cheaply repeatable on every candidate model. Their limitation is the flip side: RWK scores and BERTScore tell you about generic capability, not about your prompt templates, your system prompt, or your tool-use harness. Most teams end up pairing a one-time automatic-job comparison (model selection) with an ongoing application-level suite run through their own harness (regression testing), plus judge-model jobs where nuanced grading is needed. For completeness, Bedrock's RAG evaluation is a separate job family targeting knowledge bases and external RAG sources — covered in Bedrock RAG evaluation.
Where to go next
For judge-based jobs and their Claude judge list, see Bedrock LLM-judge setup; for budgeting, eval run costs. Vertex and Foundry equivalents are covered in Vertex partner-model evaluation and Foundry's judge-model limits.