Your existing monitoring stack already knows how to watch a service that calls Claude: request rates, error rates, and latency histograms work exactly as they do for any dependency. What it does not know is that this dependency bills by the token, varies its latency with output length, and can fail in a way no HTTP status will ever report, by returning a confident, well-formed, wrong answer. Observability for LLM apps means extending the familiar practice to cover those gaps.
The per-request record
Decide on a standard log record for every Claude call and emit it from one place (a shared client wrapper or an internal gateway) rather than trusting each team to log consistently. A workable minimum per request: a request ID that joins to your application traces; the feature or use case name; the model ID actually used; input and output token counts from the response's usage data; latency, split into time-to-first-token and total duration if you stream; the outcome (success, error class, or timeout); and retry count. With those fields you can answer the questions that actually come up: what does feature X cost per day, which feature got slower this week, and did the error spike come from one tenant or everyone.
from anthropic import AnthropicVertex
client = AnthropicVertex(project_id="my-project", region="global")
resp = client.messages.create(
model="claude-sonnet-5",
max_tokens=1024,
messages=[{"role": "user", "content": "Summarize this ticket..."}],
)
log.info("claude_call",
feature="ticket-summary",
model="claude-sonnet-5",
input_tokens=resp.usage.input_tokens,
output_tokens=resp.usage.output_tokens)Be deliberate about logging prompt and response content. Full content is invaluable for debugging and quality review, but it may contain customer data, so route it to a store with appropriate access controls and retention, or log it only for sampled or flagged requests. Metadata can go to your normal log pipeline; content usually cannot.
Latency: watch two numbers, not one
Model latency scales with output length, so a single "response time" number mixes two different stories. Time-to-first-token tells you how responsive the platform and your networking are; total duration mostly tells you how much the model wrote. Track both, as percentiles rather than averages, and segment by model tier, since Haiku 4.5, Sonnet 5, and Opus 4.8 have different performance profiles and mixing them in one histogram hides regressions in each. If you enable extended or adaptive thinking for hard tasks, expect and budget for the additional latency it introduces rather than alerting on it as an anomaly.
Errors and saturation
Break errors out by class, because they demand different responses: rate-limit errors indicate saturation against your quota and should feed capacity planning; transient server errors should be absorbed by retries and only alert when sustained; client errors indicate a bug and should alert immediately even at low volume. Also track retry rate as a first-class metric. Rising retries with stable user traffic is the earliest visible sign that you are outgrowing your limits, usually days before users notice anything.
Quality signals: the part HTTP can't see
Output quality needs its own instrumentation because no infrastructure metric moves when answers get worse. Practical signals, roughly in order of effort: explicit user feedback (thumbs up/down, edit-before-send rates); behavioral proxies (how often users retry the same question, abandon the flow, or escalate to a human); programmatic validation (does the JSON parse, do required fields appear, do cited sources exist); and sampled human review against a rubric on a fixed cadence. Log these with the same feature and model labels as everything else, so a quality dip can be lined up against a prompt change, a model version change, or a traffic shift. This is also the foundation for eval-gated model upgrades: the metrics you watch in production and the checks you run before an upgrade should be the same checks.
Where to go next
Token metrics feed directly into cost alerts and budgets and FinOps for LLMs. For turning quality signals into pre-release tests, see Evaluating LLM Features: Test Before You Trust.