Solution Patterns & Playbooks

Prompt Caching Architecture for High-Traffic Apps

At high request volume, prompt caching is the single biggest cost and latency lever you control. Getting it right is mostly a matter of treating your prompt prefix like a cache key — because that is literally what it is.

Claude 3P 101 · Updated July 2026 · Unofficial guide

Prompt caching lets Claude reuse the processed form of a prompt prefix across requests. You mark a breakpoint with "cache_control": {"type": "ephemeral"}; everything up to it is written to cache on the first request and served from cache afterwards. The economics are documented and stark: a 5-minute cache write costs 1.25x the base input price, a 1-hour write costs 2x, and every cache read costs 0.1x. On Claude Opus 4.8 at $5 per million input tokens, that is $0.50 per million for cached tokens — a 90% discount on everything behind the breakpoint.

Cache key design: the prefix is the key

Caching is a strict prefix match on the exact bytes of the rendered prompt: one changed byte at position N invalidates every breakpoint at or after N. The cache hierarchy runs toolssystemmessages, and a change at any level invalidates that level and everything after it. So architect the prompt like a cache key:

Static first, volatile last. Tool definitions and the shared system prompt go at the front; per-user and per-request content goes after the last breakpoint. The documented silent invalidators are exactly the things that sneak into "static" sections: timestamps or UUIDs in the system prompt, non-deterministic JSON serialization (set sort_keys=True), per-user tool sets, and conditional system-prompt sections. Caches are also model-scoped — a model switch forces a full rebuild — and toggling thinking modes or tool_choice invalidates the message-level cache.

You get at most 4 explicit breakpoints per request, and each breakpoint looks back at most 20 content blocks for a prior cache entry — long agentic turns that add more than 20 blocks can silently miss the cache, so place an intermediate breakpoint every ~15 blocks. For simple cases, a single top-level cache_control enables automatic caching: the API places the breakpoint on the last cacheable block and moves it forward as the conversation grows.

TTL strategy

Two TTLs exist: 5 minutes (default) and 1 hour ({"type": "ephemeral", "ttl": "1h"}). The documented break-even math makes the choice mechanical: the 5-minute cache pays for itself at 2 requests (1.25x + 0.1x beats 2x uncached), the 1-hour cache needs at least 3 hits. High-traffic chat and API endpoints re-hit shared prefixes within seconds, so 5 minutes is usually right — reads refresh at the same 0.1x rate, keeping a hot cache alive indefinitely. Reserve the 1-hour TTL for prefixes reused on a slower cadence, such as shared context in batch jobs that can take longer than 5 minutes between reuses.

Mind the minimums: prompts below a model-specific threshold silently don't cache — 512 tokens on Claude Fable 5, 1,024 on Opus 4.8, Sonnet 5, and Haiku 4.5 (and note Fable 5 requires 1,024 on Amazon Bedrock).

Cache warming

Two documented techniques matter at high traffic. First, pre-warming: send a request with max_tokens: 0 to write the cache without generating output — no output tokens are billed. A deploy pipeline that pre-warms the new system prompt means the first real user never pays the write premium. (Pre-warm requests are rejected with streaming, forced tool choice, structured outputs, or inside batches.) Second, fan-out ordering: a cache entry becomes readable only after the first response begins streaming, so parallel identical requests all pay full write price. When fanning out over shared context, await the first streamed token, then fire the rest.

Measuring hit rates in production

Every response's usage block splits input three ways: cache_creation_input_tokens (written), cache_read_input_tokens (served from cache), and input_tokens (everything after the last breakpoint). Your hit rate is cache reads divided by total input, aggregated in your observability stack; a falling ratio after a deploy is the classic signature of an accidental prefix change. Hit rate also buys throughput, not just money: cache reads do not count toward your input-tokens-per-minute rate limit on current models, so — per the documented example — a 2M ITPM limit at an 80% hit rate effectively processes 10M input tokens per minute.

Platform availability

Prompt caching works on the Claude API, Claude Platform on AWS, Google Vertex AI, and Microsoft Foundry. Amazon Bedrock supports explicit breakpoints only — no automatic caching. Cache isolation also differs: Google Cloud and Bedrock isolate caches at the organization level, the others at workspace level; confirm this against your internal tenancy model if workspaces separate customers.

Rule of thumb: if two requests in five minutes share a prefix longer than the model's minimum, cache it. If your hit rate is below ~50% on a high-traffic endpoint, hunt for a byte-level prefix instability before touching anything else.

Where to go next

See prompt cache mechanics for the request-level details, cache hit-rate strategy for the FinOps view, and multi-model routing for why caches being model-scoped matters to routers.

Sources