Cost optimizations have a way of being claimed twice and verified never. Traffic grows, models change, a promotion ends, and six months later nobody can say whether the caching project actually saved money. The antidote is a normalized metric, a baseline captured before the change, and an after-measurement on the same basis.
Normalize to cost per unit of work
Total monthly spend is the wrong yardstick — it moves with volume. Measure cost per unit of work: per ticket resolved, per document processed, per conversation. Compute it from per-request usage objects priced at your actual rates, or — on the first-party Claude API — from the Admin Usage and Cost API, which reports token consumption with breakdowns by model, workspace, and service tier, and daily costs in USD. (That API is not available on Bedrock, Vertex AI, or Claude Platform on AWS, so on those platforms your own logs plus the cloud billing console play its role.)
Capture the quality guardrail in the same baseline: an eval score, resolution rate, or review-pass rate. A savings number without a quality number next to it is unverifiable — and occasionally a regression wearing a trophy.
Isolate the change
Ship one optimization at a time per workload, and hold a comparison window on each side (one or two representative weeks, avoiding holidays and launches). Where you can, run it as an A/B split rather than before/after — time-based comparisons absorb every confounder that happened that week. Then verify with the mechanism-specific counters, because each optimization leaves its own fingerprint:
Prompt caching. The response reports cache_read_input_tokens and cache_creation_input_tokens directly. Compute effective input cost as reads at 0.1x base price plus writes at 1.25x (5-minute TTL) or 2x (1-hour) plus uncached tokens at 1x, and compare with the counterfactual of all tokens at 1x. This is the cleanest savings measurement in the whole stack because the counterfactual is exact. Sanity-check against the documented break-even: the 5-minute cache pays off at two requests per prefix; if your hit rate implies fewer, the "optimization" is a cost increase.
Model routing. Route a defined slice of traffic (say, short factual queries) from Sonnet 5 to Haiku 4.5 and measure cost per task on the routed slice only, with its own quality guardrail. Use listed prices for the counterfactual: input drops from $3 to $1 per million tokens ($2 to $1 under Sonnet 5's intro pricing through August 31, 2026), output from $15 to $5 ($10 intro). Two documented wrinkles keep this honest: token counts are model-specific (Sonnet 5's newer tokenizer produces roughly 30% more tokens than older-tokenizer models for the same text), and switching models rebuilds the prompt cache, so the first requests after a routing change pay write prices.
Output length changes. Concise-format instructions, tighter max_tokens, or a lower effort setting all show up as a drop in average output_tokens per task. Output is the expensive direction — $25 per million on Opus 4.8 versus $5 for input — so even modest trims move the total. Watch usage.output_tokens_details.thinking_tokens too: reasoning tokens are billed as output, and effort changes move them. The guardrail check matters most here, since truncation can silently damage answers (look for stop_reason: "max_tokens" in the after-window).
Keep the measurement honest over time
Three standing corrections. First, re-baseline whenever the unit price changes — Sonnet 5 moving from intro to standard pricing on September 1, 2026 will raise cost per task by the price ratio with zero behavior change, and an unadjusted dashboard will "lose" your savings. Second, attribute batch migrations separately: moving work to a 50%-priced batch mechanism is a real saving but a different lever than caching or routing, and mixing them in one number invites double-counting. Third, keep the raw usage logs. Savings claims get re-audited at budget time, and the team that can regenerate the analysis wins the discussion.
Where to go next
The natural companions are cost per quality point, cache hit rate strategy, Haiku routing, and finance-ready cost reporting to publish the results.