Calling a model over a network is a distributed-systems problem wearing an AI costume, and the classic failure modes all apply. Requests will occasionally be throttled when you exceed your platform's rate limits, time out when a long generation outlives a network intermediary's patience, or fail with a transient server-side error. None of these are emergencies if your client code expects them. All of them are emergencies if it does not.
Sort errors before you handle them
The first design decision is classification, because the right response differs by error type. Rate-limit errors (HTTP 429) mean "slow down": retrying immediately makes them worse, and retrying with backoff makes them go away. Transient server errors and dropped connections mean "try again shortly": a retry usually succeeds. Client errors (a malformed request, an unknown model ID, an oversized payload) mean "you have a bug": retrying will fail identically forever, burning quota and log space. Your error handler should treat these as three different events with three different destinations: backoff, retry, and alert-a-human respectively.
Backoff, jitter, and a retry budget
The standard pattern is exponential backoff with jitter: wait roughly one second before the first retry, double the wait each attempt, and add randomness so that a fleet of workers does not retry in lockstep and re-create the spike that caused the throttling. Cap the number of attempts, and treat that cap as a budget: three to five attempts covers genuine transients, while anything beyond that is usually masking a capacity problem you should surface instead. The anthropic Python SDK retries some transient failures for you; the sketch below shows the shape of the logic so you can reason about it, whichever layer implements it.
import random, time
from anthropic import AnthropicVertex
client = AnthropicVertex(project_id="my-project", region="global")
def call_with_backoff(messages, attempts=4):
for i in range(attempts):
try:
return client.messages.create(
model="claude-sonnet-5",
max_tokens=1024, messages=messages)
except Exception:
if i == attempts - 1:
raise
time.sleep((2 ** i) + random.random())In real code, catch the SDK's specific error classes rather than bare Exception, so client errors fail fast instead of being retried.
Retries must not duplicate work
The expensive retry bug is not the failed call; it is the successful side effect that happens twice. If a Claude response triggers an email, a ticket update, or a database write, and your system retries after a timeout whose original request actually succeeded downstream, you have sent the customer two emails. The fix is idempotency: give every logical operation a unique key, record completed operations, and make the side-effect step check the record before acting. Retry the model call freely; guard the side effect strictly. This matters doubly for tool use, where Claude itself may invoke your systems: tools that mutate state should be idempotent or protected by the same key mechanism.
Timeouts and fallbacks that degrade gracefully
Set explicit client timeouts based on what your users can tolerate, and remember that long generations legitimately take time; streaming the response (supported on all four platforms) keeps bytes flowing and avoids tripping idle timeouts in gateways and load balancers. When retries are exhausted, decide the degradation order deliberately rather than at 2 a.m. Reasonable rungs, in increasing order of engineering effort: queue the work and answer asynchronously; fall back to a different model tier that still meets the quality bar; fall back to another region; or, for multi-cloud estates, fall back to another platform entirely, which the shared SDK makes feasible but which carries real operational cost. For most teams, "queue and retry later, tell the user honestly" is the right first rung.
Finally, protect yourself from retry amplification: a circuit breaker that stops calling after a sustained failure rate, plus the cost-anomaly alerts described elsewhere in this guide, ensures a bad hour does not become a bad invoice.
Where to go next
See Streaming Responses: Better UX and Fewer Timeouts for the streaming half of this story, and Capacity Planning: Throughput, Quotas, and Peak Load for reducing how often you hit limits in the first place.