Solution Patterns & Playbooks

Idempotency and Retry Design for LLM Pipelines

Retries are where LLM pipelines quietly double-charge, double-post, and double-email. The fix is boring and reliable: classify errors correctly, retry only the retryable, and make every side effect deduplicable.

Claude 3P 101 · Updated July 2026 · Unofficial guide

The Claude API documents its error taxonomy precisely, and a robust pipeline starts by honoring it. Everything else in this pattern — back-off, dead-letter queues, exactly-once semantics — hangs off that first classification decision.

Retryable or not: the documented split

ClassCodesAction
Retryable429 rate_limit_error, 500 api_error, 504 timeout_error, 529 overloaded_error, connection errorsExponential back-off; honor retry-after on 429
Not retryable400 invalid_request_error, 401, 403, 404, 413 request_too_largeFix the request — straight to the dead-letter queue

The official SDKs already auto-retry the retryable class with exponential back-off — twice by default, honoring the retry-after header — so your queue-level retry policy sits on top of that. Retrying a 400 is pure waste: the same bytes produce the same rejection. Route non-retryable failures directly to a dead-letter queue (DLQ) — a holding queue for jobs that need a human or a code fix — along with the response body, which arrives as JSON containing the error type, message, and a request_id. Log that request_id (also present on every response as the request-id header); it is the handle Anthropic support uses to trace a request.

Python workers should branch on the SDK's typed exceptions (RateLimitError, InternalServerError, APIConnectionError, and peers) rather than string-matching messages. Two documented subtleties deserve their own handling: streaming responses can fail after the HTTP 200, with the error arriving as an in-stream event — so a "success" isn't final until message_stop; and some 400s are configuration problems wearing a request costume (for example, Claude Fable 5 rejects every request when the organization's data retention is set below 30 days). Persistent 400s on previously working code should page a human, not spin in the DLQ.

Idempotency keys

A Messages API call is not idempotent server-side — the API offers no request-deduplication parameter, and two identical calls may return two different completions. Exactly-once therefore has to be engineered at your boundaries; the practical recipe is at-least-once retries plus deduplication on a stable key:

Key the work. Derive a job key from the business input (document ID, ticket ID, content hash), and record completion in a store your workers check before calling Claude. A retried job that already completed becomes a no-op read.

Key the side effects. If the pipeline writes downstream — posts a comment, sends an email — make each write conditional on the same key, so a crash between "Claude answered" and "result recorded" cannot double-post on retry.

Two platform features hand you keys for free. In the Message Batches API, every request carries a custom_id (1–64 characters); results come back marked succeeded, errored, canceled, or expired, only successes are billed, and expired requests are not billed at all — so the safe reprocessing move is to resubmit exactly the non-succeeded custom_ids in a follow-up batch. And Managed Agents webhooks document at-least-once delivery explicitly: retries reuse the same event ID and ordering is not guaranteed, so consumers dedupe on event.id.

Back-off that respects the platform

Rate limits are token buckets that replenish continuously, and a 429 tells you precisely when capacity returns via retry-after — use it instead of a blind schedule, and add jitter so a fleet of workers doesn't retry in lockstep. Also ramp gradually after recovery: the docs note that sharp traffic spikes can trigger acceleration-limit 429s below your nominal ceiling. For 529 overloaded_error, back off more aggressively, spread requests over time, or shift traffic to a less-loaded model.

DLQ triage rule: a job lands in the dead-letter queue for exactly one of three reasons — bad input (fix the data), bad request shape (fix the code), or exhausted retries on a retryable error (re-drive after the incident). Tag each DLQ entry with its error type and request_id at enqueue time so triage is a query, not an investigation.

Platform notes

The error taxonomy above is the first-party Claude API's, shared by Claude Platform on AWS (same-day parity). Amazon Bedrock, Google Vertex AI, and Microsoft Foundry front Claude with their own service error envelopes and throttling conventions — the retry/dedupe architecture transfers unchanged, but map the specific codes from your provider's documentation. The Message Batches API (and its custom_id mechanics) is available on the first-party API and Claude Platform on AWS, not on Bedrock, Vertex AI, or Foundry.

Where to go next

This pattern pairs with queue-based async processing and retries and fallbacks; see the error-code reference for the full taxonomy.

Sources