The Claude Console — the browser workbench for the Claude API at platform.claude.com — includes an Evaluation tool that, in Anthropic's words, "allows you to test your prompts under various scenarios." It lives in the Evaluate tab of the Console prompt editor, next to where you write the prompt itself. For teams accessing Claude through Bedrock, Vertex AI, or Foundry, it's worth knowing anyway: prompts are portable, so many teams iterate in the Console and deploy the winning prompt to their cloud platform of choice.
The one prerequisite: double-brace variables
The tool has a hard requirement that trips up first-time users: your prompt must include at least one or two dynamic variables in double-brace syntax, like {{variable}}. Anthropic's docs state this plainly — "This is required for creating eval test sets." The reason is structural. A test suite is the same prompt run against many inputs; the variables mark which parts of the prompt change per test case (the customer email, the document, the question) and which parts stay fixed (your instructions). A prompt with no variables is a single hardcoded case — there's nothing to build a table of tests around. If your prompt is currently one solid block, factor the changing content into a {{variable}} first; the Console prompt generator produces templates in this shape automatically.
Three ways to fill the test table
Each row of the eval table is one test case: a value for each variable. The Console offers three ways to add rows, and most real suites use all three.
| Method | How it works | Best for |
|---|---|---|
| + Add Row | Type variable values in manually | Known tricky cases, regressions you've seen in production |
| Generate Test Case | Claude synthesizes one new row per click | Expanding coverage fast when you have few examples |
| Import CSV | Upload a file of prepared test cases | Existing datasets, production log samples, bulk edge cases |
The auto-generation is more controllable than it first appears: clicking "Show generation logic" exposes the instructions Claude uses to invent test cases, and you can edit them — Anthropic's docs describe this as customizing generation "to greater precision and specificity." If every generated row looks like a polite, average customer, edit the logic to demand angry customers, half-complete data, or non-English input. For a fuller workflow built around synthetic cases, see bootstrapping an eval set.
Running the suite and reading results
Run the suite and the Console executes your prompt once per row, collecting the outputs in a results view. Three documented features turn this from "a batch runner" into an actual evaluation loop:
Side-by-side comparison. You can compare the outputs of two or more prompts on the same test cases, in adjacent columns. This is the honest way to answer "did my rewrite help?" — same inputs, same rows, differences visible line by line rather than remembered.
Five-point quality grading. Each response can be graded on a 5-point scale in the results view. These are your judgments, recorded per cell — which makes comparisons quantitative. A rewrite that "feels better" but scores an average 3.2 against the original's 3.8 gets caught before shipping.
Prompt versioning with re-runs. When you update the prompt text, you can re-run the entire suite against the new version — Anthropic's docs highlight exactly this: see "how changes affect performance across all test cases." Create versions as you iterate and the suite becomes a regression harness: every candidate prompt faces the same panel of tests, including the edge cases that broke version one.
Where it fits — and where it stops
The Evaluation tool is manual-grading-in-a-nice-UI: excellent for prompt iteration by one person or a small team, with volume in the dozens-to-hundreds of cases. It doesn't replace a programmatic harness with code-based graders or LLM judges running thousands of cases in CI, and it grades prompts, not deployed systems. A common progression: iterate in the Console until the prompt stabilizes, export your test cases, then encode the same suite programmatically against whichever platform you deploy on.
Where to go next
The Console's three prompt tools are designed to chain — generator, improver, evaluator. See the prompt generator and the prompt improver's trade-offs, or the quickstart if you don't have Console access yet.