Most descriptions of Amazon Bedrock's evaluation feature assume a closed loop: pick a Bedrock-hosted model, Bedrock runs it over your prompts, then scores the results. But AWS documents a second mode that matters a lot for teams running Claude in more than one place. In their words: "you can evaluate a non-Amazon Bedrock model by providing your own inference response data in the prompt dataset. If you provide your own response data, Amazon Bedrock skips the model invoke step and directly evaluates the data you supply." AWS doesn't give this mode a snappy name; people commonly call it bring-your-own-inference, or BYOI.
Why this matters for Claude specifically
Claude is unusual among foundation-model families in how many doors it has: the first-party Claude API from Anthropic, Claude Platform on AWS, Bedrock itself, Google Vertex AI, and Microsoft Foundry. The models are the same, but a serious evaluation program wants to score outputs wherever they were produced — and your production traffic may not run through Bedrock at all. BYOI decouples the two halves of the job: generation happens wherever your application lives; grading happens in Bedrock.
Concrete situations where this is the right tool:
Your production Claude traffic isn't on Bedrock. If your app calls the first-party Claude API or Claude Platform on AWS, you can still capture (appropriately sampled and approved) production responses, package them into a prompt dataset, and get Bedrock's managed judging, scoring reports, and S3 output — without re-generating anything.
Offline or batch pipelines. Responses produced by a nightly batch job can be evaluated the next morning as a dataset. The generation step already happened; Bedrock only grades.
Cross-platform bake-offs. Because the grading step is identical regardless of where responses came from, BYOI gives you one consistent judge across candidates — for example, comparing the same prompts answered by Claude Opus 4.8 and by a non-Claude model, or by two different prompt versions of the same model.
How it works mechanically
You build a prompt dataset that contains both the prompts and the corresponding response data, then create an evaluation job pointing at it. Because the responses are already present, Bedrock skips invocation and moves straight to scoring. The judge side is unchanged from a normal judge-model job: you pick an evaluator model from Bedrock's documented list and metrics (built-in or custom), and results land in the console and your S3 bucket. The same bring-your-own pattern exists for RAG evaluation — both Knowledge Base job types accept "your own inference response data from an external RAG source" (see the RAG evaluation article).
A minimal harvesting loop against Claude Platform on AWS looks like this; the packaging into Bedrock's exact dataset schema should follow the format in AWS's documentation rather than anything improvised:
from anthropic import AnthropicAWS # needs AWS_REGION + ANTHROPIC_AWS_WORKSPACE_ID
client = AnthropicAWS()
rows = []
for prompt in prompts:
msg = client.messages.create(
model="claude-opus-4-8",
max_tokens=1024,
messages=[{"role": "user", "content": prompt}],
)
rows.append({"prompt": prompt, "response": msg.content[0].text})
# Convert rows to Bedrock's prompt-dataset format (see AWS eval docs),
# upload to S3, then create the evaluation job against that dataset.What stays inside Bedrock
One asymmetry to plan around: BYOI frees the generator, not the judge. The evaluator model still runs on Bedrock and must come from AWS's documented judge list — which for Claude currently tops out at Opus 4.5 / Sonnet 4.5 / Haiku 4.5, not Fable 5 or Opus 4.8 (details in the judge-setup article). That's usually fine: Anthropic's own guidance favors judging with a different model than the one that generated the output anyway.
Also remember the operational prerequisites are the ordinary Bedrock ones: an AWS account with Bedrock access, model access granted for the judge model you choose, and an S3 bucket for datasets and reports. And treat data governance as a first-class step — moving production responses from another platform into S3 for evaluation is a data flow your security team should sign off on.
Where to go next
Read the LLM-as-judge setup guide for the judge and metric options, or judge-model independence for why an older-generation judge is not a weakness. The platform overview maps the deployment surfaces this article assumes.