A tokenizer is the component that slices text into the units — tokens — that a model actually processes and that platforms bill and rate-limit by. Anthropic introduced a new tokenizer with Claude Opus 4.7, and it carried forward to Opus 4.8, Claude Fable 5, the Mythos models, and Claude Sonnet 5. For the same input text, the new tokenizer produces approximately 30% more tokens than the previous one — Anthropic's migration guide gives a range of 1x to 1.35x depending on content. Sonnet 4.6 and everything earlier still use the old tokenizer.
The practical consequence: the exact same prompt, sent at the exact same request rate, consumes up to 30% more of your tokens-per-minute (TPM) quota after you migrate from, say, Sonnet 4.6 to Sonnet 5. A workload comfortably inside its limits on the old model can start throwing 429 errors on the new one — with identical traffic.
Three budgets move at once
Quota headroom shrinks. If your pipeline consumed 1.5M input tokens per minute on Sonnet 4.6, plan for up to roughly 2M on Sonnet 5. If your platform allocation was sized just above the old consumption, the migration alone can push you over. Re-run the arithmetic from your pre-launch quota estimate with the new counts before flipping traffic.
The context window holds less text. A 1M-token window on the new tokenizer fits roughly 555,000 words of English, versus roughly 750,000 words in Opus 4.6's 1M window, per Anthropic's model documentation. Pipelines that packed documents close to the old limit may now overflow it.
Cost per request rises even where the per-token price doesn't. Opus 4.7 and 4.8 carry the same $5/$25 per-million-token list price as Opus 4.6, but the same document now contains more tokens, so the same job bills more. (All prices and ratios here come from Anthropic's published pricing and migration documentation and are subject to change.)
Measure, don't extrapolate
The ~30% figure is an average across content types; your corpus may sit anywhere in the 1x–1.35x band. Code, non-English text, and unusual formatting can tokenize differently. The only reliable approach is to recount your real prompts against the target model using the free token-counting endpoint — token counts are model-specific, and the endpoint counts with whichever tokenizer the named model uses:
from anthropic import AnthropicBedrockMantle
client = AnthropicBedrockMantle(aws_region="us-east-1")
old = client.messages.count_tokens(
model="anthropic.claude-haiku-4-5-20251001-v1:0", # old tokenizer
messages=[{"role": "user", "content": sample}])
new = client.messages.count_tokens(
model="anthropic.claude-sonnet-5", # new tokenizer
messages=[{"role": "user", "content": sample}])
print(new.input_tokens / old.input_tokens) # your real ratioOne warning from Anthropic's guidance: do not estimate Claude token counts with OpenAI's tiktoken library — it undercounts Claude tokens by roughly 15–20% on typical text, and more on code or non-English content. Stacking that error on top of a tokenizer migration produces capacity plans that are wrong twice.
A transient spike on migration day
There is a second, shorter-lived effect. Changing the model string invalidates your existing prompt cache — the first requests on the new model all write the cache fresh. Cache writes count toward input-TPM limits on every platform that meters them (cache reads generally do not). So the first minutes after cutover combine two pressures: 30% more tokens per request and a temporary loss of the cache-read exemption described in the caching-and-throughput article. A gradual traffic shift — old model to new over an hour rather than an instant — smooths both, and also stays clear of acceleration limits.
This applies identically across the Claude API, Claude Platform on AWS, Bedrock, Vertex AI, and Foundry: the tokenizer belongs to the model, not the platform. Whatever quota your platform grants, the new-generation models spend it about 30% faster per unit of text.
Where to go next
Recheck which limit binds first after re-counting — a workload that was RPM-bound can become TPM-bound purely through migration. The token-counting article covers the endpoint in detail.