No engineering team would let a database or a payment library upgrade itself underneath production. Yet teams new to LLMs routinely wire up "whatever the newest model is" and are then surprised when output formats shift, tone changes, or a carefully tuned prompt stops behaving. Models are dependencies with unusually broad behavioral surface area: even an upgrade that is better on average can be different in exactly the spot your feature depends on. The remedy is the same discipline you already apply elsewhere, adapted to how models ship.
Pin an exact model, everywhere, in config
Always specify a concrete model ID, such as claude-sonnet-5, or anthropic.claude-sonnet-5 on Bedrock, which alone among the four platforms prefixes model IDs with anthropic.. Avoid any alias-style or "default" identifier whose meaning a platform can change without you acting. Then make the pin auditable: the model ID belongs in configuration, not scattered through code, so that "which model is feature X on in production?" has a one-line answer and changing it is a config release you can roll back. If you run on more than one platform, keep a single mapping table from your internal model names to each platform's ID format; the prefix difference is a classic source of copy-paste confusion.
Log the model ID with every request as part of your standard observability record. When quality metrics move, the first question is always "did the model change?", and the log should answer it instantly.
Upgrades are releases, gated by evals
New Claude models are worth adopting; they generally bring better capability and sometimes better prices. The point of pinning is not to freeze forever, it is to make the upgrade a deliberate release with a test gate. That gate is an evaluation set: a few dozen to a few hundred real inputs from your workload with a definition of acceptable output for each, whether that is exact fields extracted, a rubric score, or a set of checks code can run. Run the eval against the candidate model, compare with the incumbent, and let the numbers decide. Without an eval set, a model upgrade is a vibes-based bet on every feature at once.
Roll forward like any risky release: dev and staging first, then a small production slice with quality and latency dashboards watched, then full traffic. Keep the old model ID in config as the rollback path, and remember that latency and cost are part of the comparison, not just output quality; a tier's pricing and performance profile can shift between generations.
Prompts and models version together
A prompt tuned against one model is only known to work against that model, so version prompts alongside the model ID they were validated with, and treat "same prompt, new model" as an untested combination. This cuts both ways: teams often re-tune prompts during an upgrade and then cannot say whether an improvement came from the model or the prompt. Change one variable at a time when you can, and record the pairing (prompt version, model version) in the same config release. The eval set makes this cheap; re-running it against a prompt tweak costs minutes.
A lightweight policy that scales
For most enterprises, the whole policy fits on a page. Every workload pins an exact model ID in config. Every workload has an eval set before it ships, however small. New model versions are evaluated within a defined window of release, upgraded behind a staged rollout when they pass, and the decision is recorded. Deprecation notices are tracked by a named owner. None of this is exotic; it is dependency management applied to the dependency most likely to change behavior silently.
Where to go next
See Upgrading Model Versions Without Breaking Production for the staged-rollout playbook, and Evaluating LLM Features: Test Before You Trust for building the eval set that gates it. Why Model IDs Differ Across Platforms covers the Bedrock prefix in detail.