The architecture is familiar: producers drop jobs onto a queue (SQS, Pub/Sub, Service Bus — whatever your cloud provides), a worker pool pulls jobs and calls Claude, results land in a store or a results topic. What makes the LLM version distinctive is the shape of the limits you are pacing against, and the fact that Anthropic ships a second, fully managed lane for bulk work: the Message Batches API. This article covers both, and when to use which.
Sizing the worker pool against real limits
Claude rate limits are enforced per organization, per model, on three axes — requests per minute (RPM), input tokens per minute (ITPM), and output tokens per minute (OTPM) — using a token-bucket algorithm that replenishes continuously rather than resetting on a fixed interval. On the documented Start tier, for example, Opus-class and Sonnet 5 traffic each get 1,000 RPM, 2M ITPM, and 400K OTPM. Two documented details change pool-sizing math: cache reads don't count toward ITPM on current models (a well-cached workload processes far more than its nominal limit), and OTPM counts only tokens actually generated — a generous max_tokens costs you nothing against the limit.
Don't unleash the full pool at once: the docs warn that sharp usage spikes can trigger 429s from acceleration limits even below your ceiling, so ramp traffic gradually after deploys and scale-ups.
Back-pressure: let 429 drive the queue
When a limit is exceeded, the API returns a 429 naming the exceeded limit plus a retry-after header stating how many seconds to wait. Every response also carries anthropic-ratelimit-* headers with your limit, remaining budget, and reset time — showing the most restrictive limit currently in effect. The recommended wiring: workers treat retry-after as authoritative (re-queue the job with that delay rather than hammering), and an autoscaler watches the remaining-token headers to shrink or grow the pool ahead of the 429s. 429s, 5xx errors, and connection drops are retryable; the official SDKs already retry them with exponential backoff, honoring retry-after. A 529 overloaded_error means API-wide load — back off harder, and consider routing that job to a less-loaded model.
One more timing constraint shapes worker design: for requests that would run past ~10 minutes, use streaming — SDKs validate that non-streaming requests won't exceed a 10-minute timeout.
The managed lane: Message Batches
If jobs can tolerate hours of latency, skip your own pacing entirely. The Message Batches API accepts up to 100,000 requests or 256 MB per batch, charges 50% of standard prices across all models, and finishes most batches in under an hour; anything unfinished at 24 hours expires unbilled. Each request carries a unique custom_id, results stream back as a .jsonl file from the batch's results_url (downloadable for 29 days), and per-request results are marked succeeded, errored, canceled, or expired — only successes are billed. Batches get their own rate limits (Start tier: 1,000 RPM and 200,000 requests in the processing queue), separate from your real-time traffic — which is precisely the fan-out pattern: real-time lane for users, batch lane for the nightly million, neither starving the other.
from anthropic import AnthropicAWS # Claude Platform on AWS
client = AnthropicAWS()
batch = client.messages.batches.create(requests=[
{"custom_id": f"doc-{d.id}",
"params": {"model": "claude-haiku-4-5", "max_tokens": 512,
"messages": [{"role": "user", "content": d.text}]}}
for d in documents
])
# Poll batches.retrieve(batch.id) until processing_status == "ended",
# then stream the .jsonl at results_url and join on custom_id.Practical guidance
Use the batch's custom_id as your job ID so results join back to queue messages deterministically. Because batches can run longer than 5 minutes, use the 1-hour prompt-cache TTL for context shared across a batch. Keep per-job payloads within the documented request-size limits (32 MB for Messages, 256 MB for a whole batch). And log the request-id header from every real-time call — it is what Anthropic support will ask for.
Where to go next
Pair this with idempotency and retry design for the failure paths, the Batch API in detail, and batch vs. realtime for the workload-placement decision.