Retrieval-augmented generation (RAG) — feeding a model documents fetched from your own knowledge store so its answers are grounded in your data — is the most common enterprise Claude architecture. It is also the hardest to evaluate by eyeball, because a fluent answer can hide a bad retrieval, and a good retrieval can be squandered by a bad synthesis. Amazon Bedrock ships a managed evaluation type for exactly this: Knowledge Base (RAG) evaluation jobs, which AWS positions as a way to "compare different Amazon Bedrock Knowledge Bases and other RAG sources, and then to choose the best Knowledge Base or RAG system for your application."
Two job types for two failure modes
| Job type | What is scored | Use it when |
|---|---|---|
| Retrieve only | A report based on the retrieved data itself | You're tuning chunking, embeddings, or search configuration and want to isolate retrieval quality |
| Retrieve and generate | Retrieved data plus the generated summaries/answers | You want the end-to-end verdict users actually experience |
Running both is often worth it: if end-to-end scores drop but retrieve-only scores hold steady, the regression is in the generation half (prompt, model choice, or generation settings), not your index.
The ground-truth requirement
These evaluations are LLM-based — a judge model computes the metrics — but they are not free-form vibes. AWS is explicit that the dataset must include ground truth: the expected retrievals and expected responses for each query. The evaluations then "reveal if a RAG source or Amazon Bedrock Knowledge Base is able to retrieve highly relevant information and generate useful, appropriate responses" relative to that reference. Practically, this means the expensive part of RAG evaluation is not the job itself but curating a query set with agreed-correct answers and source passages. Anthropic's evaluation guidance applies here: mirror your real query distribution, include edge cases (queries with no answer in the corpus, ambiguous queries), and prefer volume of automatable test cases over a handful of hand-graded ones.
Claude's two roles: generator and judge
For retrieve-and-generate jobs, the response generator can be any Bedrock foundation model — which includes the Claude family — as well as Marketplace, customized, or imported models, prompt routers, and Provisioned Throughput models. So a Knowledge Base fronted by Claude Opus 4.8 or Sonnet 5 is squarely in scope as the thing being tested.
The judge side is more constrained. AWS's documented RAG evaluator models explicitly include Anthropic Claude — Claude 3.5 Sonnet v1/v2, Claude 3.7 Sonnet, Claude 3 Haiku, Claude 3.5 Haiku, Claude Haiku 4.5, Claude Opus 4.5, and Claude Sonnet 4 on the built-in metrics list — alongside Amazon Nova, Meta Llama 3.1 70B, and Mistral Large, with a similar Claude set for custom metrics.
Bringing a non-Bedrock RAG source to the bake-off
Both job types allow you to "bring your own inference response data from an external RAG source" instead of pointing at a Bedrock Knowledge Base. That makes this feature a neutral referee, not just a Bedrock self-check. If you're weighing a Bedrock Knowledge Base against a self-built pipeline — say, your own vector database with Claude called through Claude Platform on AWS or the first-party API — you can run your external pipeline offline, capture its retrievals and responses into the dataset format, and have the same judge score both systems on the same queries. The mechanics mirror bring-your-own-inference for model evaluation: generation happens wherever it happens; grading is centralized in Bedrock.
Two practical cautions. First, keep the judge, metrics, and query set fixed across all candidates — change one variable at a time or the comparison tells you nothing. Second, follow AWS's documented dataset schema for supplying external retrievals and responses rather than inventing fields; the userguide pages below are the authority on the exact format.
Where to go next
For the non-RAG evaluation job types, see LLM-as-judge setup and Bedrock automatic evaluation jobs. If you're still choosing where your RAG stack should live, start from the platform overview.