The basic internal AI gateway centralizes credentials and logging. This article covers the second wave of features teams bolt on: per-team limits, fallback routing, and policy enforcement. The gateway logic itself is your code (or an API-management product you already run); what follows anchors each extension to documented Claude API behavior.
Per-team limits: use the platform first
Before writing your own quota logic, use what the platform gives you. On the Claude API, limits are enforced per organization per model as requests per minute (RPM), input tokens per minute (ITPM), and output tokens per minute (OTPM), using a token-bucket algorithm that replenishes continuously. Workspaces can carry their own custom spend and rate limits set below the organization's (not on the default workspace), and a Rate Limits API reads the configured limits programmatically — so "one workspace per team, one gateway route per workspace" gets you real isolation with no bookkeeping code.
Two documented subtleties are worth encoding in gateway math. First, ITPM is cache-aware on most models: cache reads don't count, so a workload with an 80% cache hit rate on a 2M ITPM limit effectively processes 10M input tokens a minute — a gateway that throttles on raw tokens will kick in far too early. Second, OTPM counts only actually generated tokens, so a generous max_tokens costs nothing in rate-limit terms.
Backpressure: mirror the headers, honor the hints
When a limit is exceeded, the API returns 429 naming the exceeded limit plus a retry-after header. Every response also carries anthropic-ratelimit-{requests,tokens,input-tokens,output-tokens}-{limit,remaining,reset} headers reflecting the most restrictive limit currently in effect. A good gateway does three things with these: propagates retry-after to callers, queues rather than retries when headroom is low, and ramps new workloads gradually — sharp spikes can trigger acceleration-limit 429s even below your steady-state ceiling.
Fallback routing
Route around three failure classes differently:
- 429 rate limits: limits are per model, and the buckets are documented — Opus 4.8/4.7/4.6/4.5 share one combined bucket while Sonnet 5 has its own — so falling back from an exhausted Opus bucket to Sonnet 5 buys genuinely separate headroom. Falling back Opus-to-Opus does not.
- 529 overloaded: reflects API-wide load; documented options are backoff, a less-loaded model, or spreading requests over time.
- Platform outage: cross-platform fallback (say, Bedrock to Vertex) is a bigger hammer — model IDs, feature availability, and auth all differ, so pre-test the degraded path. See multi-platform failover.
Remember that official SDKs already retry 429/5xx/connection errors twice with exponential backoff; gateway-level retry on top of SDK retry multiplies traffic during incidents. Pick one layer to own retries. And any fallback that switches models pays a documented tax: prompt caches are model-scoped, so the first requests after a switch run uncached.
Policy enforcement at the choke point
Because every request passes through it, the gateway can enforce house rules deterministically:
- Model allowlist and pinning: reject unknown model IDs; normalize per-platform forms (Bedrock's
anthropic.prefix vs. bare IDs elsewhere). - Parameter hygiene: strip or reject parameters the target model refuses — for example
temperatureon Fable 5 or Opus 4.8 returns a 400 — so teams get a clear gateway error instead of a confusing API one. - Tool policy: require
strict: trueon tool definitions where schema fidelity matters, or forcetool_choicefor constrained endpoints. - Size guards: the Messages API caps requests at 32 MB (413
request_too_large); rejecting oversized payloads at the gateway gives faster, cheaper failures.
Where to go next
Feed gateway decisions from the observability stack, and use the gateway's traffic-split hook for blue/green model rollouts. For chargeback on top of per-team routing, see the chargeback model.