Anthropic's evaluation docs acknowledge the bootstrapping problem directly: "Writing hundreds of test cases can be hard to do by hand! Get Claude to help you generate more from a baseline set of example test cases." The Console operationalizes that advice with a test case generator built into the Evaluation tool: from a seed example or two, Claude synthesizes new input rows on demand, and you shape them into a labeled dataset before your first real user ever shows up.
Step 1: seed and generate inputs
The workflow starts in the Console prompt editor with a templated prompt — one using {{variable}} placeholders, which the Evaluation tool requires. Add a row or two by hand representing realistic inputs: an actual support email, a genuine document excerpt. These seeds anchor the generator in reality. Then click Generate Test Case, and Claude produces a new row of synthetic variable values each time — Anthropic's prompting-tools docs describe the same move as the way to build examples when you don't have any: generate sample inputs, then collect Claude's responses to them.
The quality lever most users miss is "Show generation logic." The instructions Claude follows when inventing test cases are visible and editable — the docs frame this as tuning generation "to greater precision and specificity." Out of the box, generated cases tend toward the plausible middle of the distribution. Edit the logic to demand what the middle lacks: unusually long inputs, missing fields, hostile tone, irrelevant content, multiple languages — the categories from Anthropic's own documented edge-case list (irrelevant or nonexistent input data, overly long input, poor or harmful user input, and ambiguous cases). See the edge-case taxonomy for a fuller treatment.
Step 2: turn Claude's responses into ideal answers
Synthetic inputs are only half a dataset; an eval needs the expected outputs too — often called golden answers. Anthropic's documented shortcut: run the inputs through Claude, then edit the responses into ideal outputs. Reviewing and correcting a draft answer is dramatically faster than authoring from scratch, and the editing step is precisely where your judgment enters the dataset. The docs describe this workflow for building polished examples — generate inputs, get Claude's responses, edit them into ideal outputs — and it applies identically to golden answers: fix errors, tighten format, make each answer the response you'd want at 2 a.m. from your production system.
Step 3: assemble, grade, and keep the set alive
With inputs and ideal answers in place you have a labeled dataset, and everything downstream unlocks. Inside the Console, run the suite, compare prompt versions side by side, and grade on the built-in 5-point scale — the full loop covered in the Evaluation tool walkthrough. Outside the Console, the same rows drive a programmatic harness: exact-match and string-match checks against your golden answers (code-based grading) or rubric-driven LLM-as-judge scoring, on whichever platform you deploy — the Console also accepts CSV import if you assemble the dataset elsewhere first, and platform eval services like Amazon Bedrock's take custom prompt datasets in the same spirit.
Two habits keep a bootstrapped set honest over time. First, mind the volume principle — Anthropic explicitly prefers more test cases with slightly lower-signal automated grading over a handful of hand-polished ones, so keep clicking generate: dozens to hundreds of cases, not five artisanal ones. Second, treat synthetic data as scaffolding with a replacement schedule. Generated cases share the generator's idea of "typical," and real users are reliably stranger. Once production logs exist, fold real (appropriately handled) queries into the suite and let them gradually outnumber the synthetic seeds — keeping the synthetic edge cases, which remain valuable precisely because they're rare in the wild.
Where to go next
Define what "pass" means before generating anything — SMART success criteria — then see the prompt generator for getting the templated prompt this workflow starts from, and volume over quality for the strategy behind scaling the set.