An "eval" is a test case for model behavior: a fixed input, the model's output, and a scoring rule. Anthropic's own prompt-engineering guidance lists clear success criteria and empirical tests against them as prerequisites before you even start tuning prompts. The CI pattern operationalizes that: every change to a prompt, tool schema, or model ID triggers the suite, and the merge is blocked if scores drop.
The pipeline shape
Three stages, all orchestrated by ordinary CI tooling (GitHub Actions, GitLab CI, Jenkins — your choice):
- Generate: run every eval input through the candidate configuration and collect outputs.
- Score: apply deterministic checks in code where possible (exact match, regex, JSON validity, "did it call the right tool"), and an LLM-as-judge only for genuinely subjective qualities like tone or faithfulness.
- Gate: compare scores to a stored baseline and fail the build on regression. The gate itself is plain code — no model in the loop.
Run the suite as a batch
Eval suites are the textbook case for the Message Batches API: hundreds or thousands of independent requests, no latency requirement. All batch usage is charged at 50% of standard prices, most batches finish in under an hour, and batches that don't complete within 24 hours expire with unprocessed requests not billed. Each request carries a custom_id (1–64 characters) — use your eval-case ID so results join back to inputs — and results stream back as a JSONL file from the batch's results_url, downloadable for 29 days. Only requests with result type succeeded are billed; your harness should count errored and expired results and retry or fail loudly rather than silently shrinking the denominator.
Platform availability matters here. The Message Batches API is available on the first-party Claude API and Claude Platform on AWS, but not on Amazon Bedrock, Google Vertex AI, or Microsoft Foundry. Bedrock and Vertex have their own cloud-native batch inference for Claude (S3-based and BigQuery/GCS-based respectively); alternatively, many teams run CI evals against a 1P or Claude Platform on AWS key even when production traffic lives elsewhere — just remember the two surfaces can differ in available features.
LLM-as-judge, made auditable
When a judge model scores outputs, force the verdict into a schema so the gate can parse it. Structured outputs are generally available on the Claude API, Claude Platform on AWS, Bedrock, and Vertex AI (beta on Foundry):
from anthropic import AnthropicVertex
schema = {"type": "object", "additionalProperties": False,
"required": ["score", "reasoning"],
"properties": {"score": {"type": "integer"},
"reasoning": {"type": "string"}}}
client = AnthropicVertex(project_id="my-proj", region="global")
resp = client.messages.create(
model="claude-opus-4-8", max_tokens=1024,
messages=[{"role": "user", "content": rubric_and_output}],
output_config={"format": {"type": "json_schema", "schema": schema}},
)Two caveats from the structured-outputs docs: numeric range constraints such as minimum/maximum are not supported in schemas (enforce the 1–5 range in code), and a refusal comes back as HTTP 200 with stop_reason: "refusal" — treat that as a scoring failure, not a zero. The Python SDK also offers a client.messages.parse() helper that validates responses against Pydantic models client-side.
Keeping CI costs sane
Batch pricing already halves the bill (for example, Opus 4.8 drops to $2.50/$12.50 per million input/output tokens in batches). Beyond that: the token-counting endpoint is free and estimates suite cost before you run it; a smaller fast model like Haiku 4.5 ($0.50/$2.50 batched) handles deterministic-adjacent checks; and batch rate limits are generous but real — at the Start tier, 200,000 requests can sit in the processing queue. Run the full suite on merges to main and a smaller smoke slice on every pull request.
Where to go next
This gate is what makes blue/green model rollouts safe, and it leans on the same batch machinery you'd use for archive processing. For scoring-prompt design, see the prompt evaluation framework.