Everything you are billed for, rate-limited on, and context-constrained by is denominated in tokens — the chunks of text a model reads and writes. What ordinary documentation glosses over is that the mapping from text to tokens is a property of the model's tokenizer, and tokenizers change between generations. The same 10,000-word contract is a different number of tokens on Claude Sonnet 4.6 than on Claude Sonnet 5. That is tokenizer drift, and it invalidates a surprising range of numbers your systems may have hard-coded.
The current fault line in the Claude lineup
Anthropic's pricing documentation states it directly: Claude Opus 4.7 and later Opus models, Fable 5, Mythos 5, Mythos Preview, and Sonnet 5 use a newer tokenizer that produces approximately 30% more tokens for the same text — the exact increase depends on the content — while Sonnet 4.6 and earlier models use the previous tokenizer. The Opus 4.6 → 4.7 migration guide gives the observed range as 1x to 1.35x more tokens for the same input.
The effect on effective context capacity is concrete. On the new tokenizer, a 1M-token window holds roughly 555k words (~2.5M unicode characters); on Opus 4.6's tokenizer, the same nominal 1M window held roughly 750k words (~3.4M characters). Same advertised window, about a quarter less text. Conversely, some upgrades involve no drift at all: Claude Fable 5 uses the same tokenizer introduced with Opus 4.7, so Opus 4.8 → Fable 5 token counts are roughly unchanged.
What silently breaks
| Number you measured on the old model | What drift does to it |
|---|---|
| Cost-per-request projections | Understated — both input and output counts rise |
| Context-budget arithmetic ("our prompt uses 40k of 200k") | Wrong headroom; truncation logic fires late or not at all |
| Chunk sizes in RAG pipelines sized in tokens-per-chunk | Chunks overshoot their target size |
| Rate-limit and quota planning (tokens per minute) | The same traffic consumes quota faster |
| Prompt-caching breakpoints near the cache minimum | Boundary assumptions shift |
The quota point deserves emphasis on third-party platforms: Bedrock, Vertex, and Foundry throughput allocations are token-denominated, so a ~30% token inflation is effectively a ~30% haircut on how much text your existing quota moves (see tokenizer changes and quota).
Why you cannot reuse old measurements
Anthropic's migration guide makes re-measurement an explicit checklist item: "Re-run token counting against Claude Sonnet 5 rather than reusing counts measured against earlier models." The instruction generalizes. A token count is a measurement made with a specific instrument; quoting a count without naming the model it was measured against is like quoting a length without units. Note also that the drift is content-dependent — a 30% average can be higher or lower for your particular mix of code, prose, or non-English text — which is exactly why the guide says to re-run counts on your data rather than apply a blanket multiplier.
Building a token-count regression step
The fix is mechanical, which means it belongs in your upgrade process, not in someone's memory. Keep a fixed corpus of representative payloads — real prompts, real documents, your longest cases — and on every candidate model, count them via the platform's token-counting support and diff against the stored baseline:
from anthropic import AnthropicVertex
client = AnthropicVertex(project_id="my-project", region="global")
def count(model: str, prompt: str) -> int:
result = client.messages.count_tokens(
model=model,
messages=[{"role": "user", "content": prompt}],
)
return result.input_tokens
drift = count("claude-sonnet-5", corpus) / count("claude-sonnet-4-6", corpus)A drift ratio near 1.0 means measurements carry over; a ratio like 1.3 means every downstream number — budgets, chunk sizes, truncation thresholds, cost forecasts — needs re-derivation before the model swap ships. Store the ratio in your migration notes alongside the re-baselined cost figures. Token counting is available across the platforms (generally available on the Claude API, Claude Platform on AWS, Bedrock, and Vertex AI; beta on Foundry), so the same regression step ports wherever you run.
Where to go next
Tokenizer drift is step two of the migration regression checklist. For the mechanics of counting on each platform, see token counting, token counting on Vertex, and tokenizer breaks on upgrade.