Anthropic's evaluation docs make a point that single-metric leaderboards obscure: "Most use cases will need multidimensional evaluation along several success criteria." The docs list the recurring dimensions by name — task fidelity, consistency, relevance and coherence, tone and style, privacy preservation, context utilization, latency, and price. Very few production systems can ignore more than a couple of these. This article looks at the three that trade off against each other most directly — fidelity, latency, and cost — and how to decide which one wins when they conflict.
Why one number is never enough
Imagine a customer-support classifier that hits 96% accuracy in testing. Ship it? You can't know. If it reaches that accuracy by running a large model with a long reasoning budget, median response time may be unacceptable for a live-chat flow. If each classification costs several cents, the economics may collapse at your ticket volume. And if it's accurate on average but inconsistent — the same ticket classified differently on Tuesday than on Monday — downstream automation breaks in ways an accuracy score never reveals.
Each of those failure modes lives on a different dimension, and each needs its own SMART criterion: an accuracy or F1 threshold for fidelity, a response-time bound in milliseconds for latency, a per-request or per-resolved-ticket cost ceiling for price. Anthropic's docs explicitly include "response time (ms)" and price among the example metrics — they are first-class evaluation targets, not afterthoughts.
The three-way trade-off, concretely
On any platform, the levers that raise fidelity — a more capable model, longer prompts with more examples, extended or adaptive thinking, retries and best-of-N sampling — almost all raise latency, cost, or both. The model tier alone spans an order of magnitude: at July 2026 list prices, Claude Haiku 4.5 is $1 per million input tokens and $5 per million output tokens, while Claude Fable 5 is $10 and $50. Those list prices match across the Claude API and the cloud marketplaces, so the trade-off shape is the same whether you buy through Bedrock, Vertex AI, or Foundry.
| Dimension | Example criterion | Typical lever |
|---|---|---|
| Task fidelity | F1 ≥ 0.85 on a held-out test set | Model tier, prompt examples, thinking budget |
| Latency | p95 response time under a fixed ms budget | Smaller model, shorter outputs, streaming |
| Cost | Cost per request below a ceiling at projected volume | Model tier, prompt caching, cloud-native batch |
Measure all three in the same eval run. A prompt change that lifts fidelity two points while doubling output length (and therefore cost and latency) should be visible as exactly that — a trade, not a win.
Deciding which dimension gets the weight
There is no universal ranking; there is a use-case-driven one. Three questions settle most disputes:
Is a human waiting on the answer? Interactive flows — chat, autocomplete, agent steps a user watches — put latency near the top. Pipelines that run overnight put it near the bottom, and can exploit the cloud-native batch options (S3-based on Bedrock, BigQuery/GCS-based with 50% pricing on Vertex AI) where latency is irrelevant by design.
What does one error cost? When a wrong answer triggers a refund, a compliance incident, or a bad medical suggestion, fidelity dominates and you pay for it. When errors are cheap and recoverable — a mislabeled tag someone fixes later — fidelity can yield to cost.
Does the unit economics have a hard ceiling? If the feature earns a fixed amount per use, cost per request has a mathematical maximum. Establish it first; it prunes the model and prompt options before any quality testing starts.
Running it in practice
Fidelity comes from code-based graders or LLM judges. Latency and cost come free with every API call — capture response time and token usage from the responses in the same harness, and report per-dimension results side by side. Re-run the full suite whenever you change prompts, and especially on model migrations: Anthropic's migration guide repeats "re-baseline cost and latency on your own workloads" for every documented migration path, because a new model shifts all three dimensions at once, not just quality.
Where to go next
See the Console Evaluation tool walkthrough for a low-lift way to compare prompt variants, and the platform overview for where each cloud's eval tooling fits.