Most production prompts start life as a plausible first draft that works on the three examples someone tried in a playground. What happens next separates teams: some ship it and firefight anecdotes forever; others run a refinement loop until the prompt holds up across the real distribution of inputs. Anthropic's own prompt-engineering documentation is explicit about the entry ticket: before you engineer a prompt, you need a clear definition of success, empirical tests against that definition, and a first draft to improve. This article turns that into a working method you can run on any platform — Bedrock, Vertex AI, Foundry, or Claude Platform on AWS — since the loop is about process, not API surface.
Step 1 — Draft against explicit success criteria
Write down what "good output" means before writing the prompt: required content, format, tone, length, and the failure modes you must not see. Then draft using the established techniques — clear and direct instructions, a role in the system prompt, XML-tagged structure, and 3–5 diverse examples where format consistency matters. The draft's job is not to be perfect; it is to be testable.
Step 2 — Test on a set, not an anecdote
Collect 20–50 representative inputs, deliberately including the ugly ones: long inputs, ambiguous inputs, inputs missing information, inputs in the wrong format. Run the prompt across all of them and grade against your criteria. Two practical notes for the third-party platforms: token counting is free (with its own rate limits) if you want to track prompt-size growth as you iterate, and if your test set gets large, batch-style processing at 50% pricing can cut evaluation costs — via Anthropic's Message Batches API on Claude Platform on AWS, or the cloud-native batch mechanisms on Bedrock and Vertex AI.
Step 3 — Diagnose by failure category, not by example
Resist fixing individual bad outputs. Sort failures into categories first — format drift, ignored instruction, hallucinated content, wrong scope, over-length — and fix the biggest category with one targeted change. Common diagnosis-to-fix mappings:
| Symptom | Likely fix |
|---|---|
| Instruction followed sometimes | Make it explicit and add the reason it matters — Claude generalizes from explanations |
| Format varies across inputs | Add diverse <example> pairs holding the output shape rigid |
| Model confuses instructions with input | Separate sections with XML tags |
| Long-document answers miss the point | Move documents to the top, query to the end — worth up to 30% on quality |
Step 4 — Change one thing, re-run everything
The loop only converges if each iteration is a controlled experiment: one change, full re-run, compare scores. Re-running everything matters because prompt fixes regress other cases with depressing regularity — the instruction that fixed your verbosity problem may have suppressed detail the extraction cases needed. Keep every prompt version in version control with its score, so "which version was better" is a lookup, not an argument. (That discipline has its own article: treating prompts like code.)
Know when the prompt isn't the problem
Anthropic's overview makes a point teams often miss: not every failing evaluation is a prompting problem. If the failures are latency or cost, a different model is often the easier fix than another prompt iteration — dropping to a faster tier, or stepping up a tier and simplifying the prompt. Similarly, if you've been through the loop several times and a capability just isn't there, escalating model tier is a legitimate move; see prompting differences across Haiku, Sonnet, and Opus. And some requirements shouldn't be prompt-enforced at all — guaranteed JSON belongs to structured outputs, not another round of "please return valid JSON."
Tooling that shortens the loop
The Claude Console (on the first-party platform) ships a prompt generator, templates with variables, and a prompt improver — useful for producing a stronger starting draft and for mechanizing variable substitution across your test set. Whatever platform serves production, nothing stops you from developing and testing the prompt text with these tools; prompts are portable strings. The exit condition for the loop is your success threshold on the full test set — after which the same set becomes your regression suite for model upgrades, covered in regression testing prompts after model upgrades.
Where to go next
Build the measurement side with a prompt evaluation framework, then see A/B testing prompts in production for refinement after launch.