A classifier looks deceptively simple: text in, label out. The failure modes are all in the details — labels the model interprets differently than you do, outputs that don't parse, and accuracy that drifts silently as your inputs change. This pattern walks through the four pieces of a production classification service: taxonomy, output contract, calibration, and monitoring. The Claude features involved are ordinary Messages API capabilities available on all four third-party platforms, so this pattern is unusually portable.
Design the taxonomy for the model, not the org chart
Most classification failures are taxonomy failures. Follow the documented prompting guidance: treat Claude like a brilliant but new employee — if a colleague with minimal context would confuse two of your labels, Claude will too. Give each label a one-sentence definition and, where labels are commonly confused, say explicitly what does not belong. Include 3–5 diverse worked examples wrapped in <example> tags; the official guidance calls few-shot examples one of the most effective techniques available. Keep an "other/unknown" label so the model has a legitimate exit instead of forcing a bad match.
Make the output a contract
Never parse a label out of prose. Two documented mechanisms give you a guaranteed shape. First, structured outputs: set output_config to a JSON schema whose label field is an enum, and the response is guaranteed to parse. Second, strict tool use: define a single tool with strict: true and force it with tool_choice: {"type": "tool", "name": ...} — tool inputs are then guaranteed schema-valid. Note one documented interaction: forced tool choice is incompatible with extended thinking, so for a thinking-enabled classifier prefer structured outputs. Also note that prefilling the assistant turn — an old classification trick — is no longer supported from the Claude 4.6 generation onward and returns a 400 error; structured outputs are the documented replacement.
from anthropic import AnthropicBedrockMantle
client = AnthropicBedrockMantle(aws_region="us-east-1")
schema = {"type": "object", "additionalProperties": False,
"properties": {
"label": {"type": "string", "enum": ["billing", "bug", "account", "other"]},
"confidence": {"type": "string", "enum": ["high", "medium", "low"]}},
"required": ["label", "confidence"]}
msg = client.messages.create(
model="anthropic.claude-haiku-4-5-20251001-v1:0", max_tokens=256,
output_config={"format": {"type": "json_schema", "schema": schema}},
messages=[{"role": "user", "content": ticket_text}])
stop_reason: "refusal" and may not match your schema — check it before trusting the payload.Calibrate before you automate
The self-reported confidence field above is a useful routing signal, but treat it as uncalibrated until proven otherwise. The recommended practice: hold out a few hundred human-labeled examples, run the classifier, and measure accuracy per label and per confidence band. That tells you where "high" confidence is actually trustworthy, which becomes your automation threshold — high-confidence labels flow straight through; everything else routes to a human queue (see the escalation pattern). Anthropic's prompt-engineering guidance is explicit that clear success criteria and empirical tests against them are prerequisites, not afterthoughts.
Haiku 4.5 ($1 input / $5 output per million tokens) is the natural starting model for short-text classification; move up to Sonnet 5 only if your eval shows the accuracy gap justifies the price. For backfilling historical data, batch processing halves the price — available via Anthropic's Message Batches API on the Claude API and Claude Platform on AWS only, while Bedrock and Vertex AI offer their own cloud-native batch inference mechanisms instead.
Monitor for drift, not just errors
Once live, deterministic code — not the model — should own the operational loop (retries on 429/5xx, timeouts, schema validation). What the model can't tell you is that your inputs changed. Recommended practice: log every prediction with its input hash, model ID, and prompt version; sample a fixed percentage into ongoing human review; and alert on distribution shifts — a sudden rise in "other" or "low confidence" usually means new input types, not a broken model. Re-run your held-out eval whenever you change the prompt, the taxonomy, or the model ID, and version prompts like code so you can roll back.
Where to go next
Pair this with escalation to human for the low-confidence path and extraction to database when you need fields, not just labels. The platform overview covers where each feature runs.