Claude costs regress the way performance regresses: quietly, one well-intentioned change at a time. Someone pastes three more few-shot examples into the system prompt. A tool schema gains ten verbose parameter descriptions. A retrieval step starts returning twenty chunks instead of five. None of these break tests — every one of them raises the price of every request, forever. The fix is the same as for performance: measure in CI, compare against a baseline, and fail the build when the number jumps.
The free instrument: the token counting endpoint
Anthropic's token counting endpoint (POST /v1/messages/count_tokens) accepts exactly what message creation accepts — system prompts, tools, messages, images, PDFs — and returns {"input_tokens": N} without running the model. It's free to use, with its own generous rate limits (2,000 requests per minute at the Start tier) that are separate from your message-creation limits, so CI runs never compete with production traffic. It is supported across the platforms, so the same check works whether production calls Bedrock, Vertex AI, Foundry, or the Claude API.
The endpoint is stateless — to diff two versions of a prompt, count each and subtract. A minimal CI gate:
from anthropic import AnthropicVertex
client = AnthropicVertex(project_id="my-project", region="global")
def prompt_tokens(system: str, tools: list) -> int:
resp = client.messages.count_tokens(
model="claude-sonnet-5",
system=system,
tools=tools,
messages=[{"role": "user", "content": "representative input"}],
)
return resp.input_tokens
baseline = 4200 # checked into the repo, updated deliberately
current = prompt_tokens(NEW_SYSTEM_PROMPT, TOOLS)
assert current <= baseline * 1.10, f"{current} tokens vs baseline {baseline}"Swap the client for your platform (AnthropicBedrockMantle with anthropic.claude-sonnet-5 on Bedrock, and so on) — the counting logic is identical.
What to put under test
The fixed prefix. System prompt plus all tool definitions, counted against a small set of representative user inputs. This is the number multiplied by every single request, so it deserves the tightest threshold — 10% growth is a reasonable default gate, with a hard absolute ceiling as backstop.
Per-fixture end-to-end counts. For pipelines that assemble context (RAG chunks, conversation history, retrieved documents), count the fully assembled request for a handful of frozen fixtures. This catches the "retrieval now returns 4x the chunks" class of regression that prompt-file diffs miss.
Output-side spend, via evals you already run. Output tokens can't be pre-counted — they depend on what the model generates, and output is the expensive direction ($15 vs. $3 per million on Sonnet 5 at standard rates; thinking tokens also bill as output). So attach cost assertions to your existing eval suite: record usage.output_tokens (and output_tokens_details.thinking_tokens where present) per eval case and alert on meaningful growth in the aggregate. A max_tokens change is a cost-relevant diff too — review it like one.
Three baseline gotchas
Counts are model-specific. Claude Opus 4.7 and later, Fable 5, and Sonnet 5 use a newer tokenizer that produces roughly 30% more tokens than earlier models for the same text. A model upgrade legitimately moves the baseline — recount against the model you'll actually run, and treat "model changed" as an intentional baseline reset, not a regression.
Counts are estimates. Actual input tokens at message-creation time may differ slightly, and system-added tokens (like the tool-use system prompt) may appear in counts without being billed identically. Thresholds, not exact-match assertions.
Count the cache-adjusted number too. A prompt refactor can keep total tokens flat while destroying the stable prefix that made caching work — same count, several times the cost, since cache reads bill at 0.1x base input. If you rely on caching, also assert that the intended breakpoint prefix is byte-stable across your fixtures (see maximizing cache hit rates).
Where this fits in the pipeline
Token gates are pre-deployment; they complement, not replace, runtime anomaly detection, which catches volume-driven spikes no CI check can see. Together they bracket the two ways bills explode: per-request cost (CI) and request count (production monitoring). When a gate fails legitimately — the feature genuinely needs more context — update the baseline in the same PR and let the diff double as a budget notification to whoever owns the feature's forecast.
Where to go next
Shrink the numbers the gate protects with a system prompt audit and tool schema optimization.