Observability for an LLM workload answers three questions on demand: is it healthy (latency, errors), what is it costing (tokens, by which workload), and is it behaving (refusals, truncations, quality signals). On Claude Platform on AWS there is an extra reason to take this seriously: the programmatic Usage and Cost Admin API endpoints available on the first-party Claude API are not available here — the Claude Console shows per-workspace usage and cost pages, and billing arrives through AWS Marketplace metered in Claude Consumption Units (CCUs, $0.01 each), invoiced monthly in arrears. Fine for finance; too slow and too coarse for engineering. Your structured logs and metrics are the near-real-time record.
Structured logs: one event per call
Emit a single JSON log event per Claude call with a stable schema. The non-negotiable fields: both request IDs — x-amzn-requestid (AWS-side, indexed in CloudTrail, quote it to AWS support) and request-id (Anthropic-side, quote it to Anthropic support) — plus the model ID, workspace ID, token counts from the response's usage object, latency, stop_reason, and your own correlation fields (feature name, prompt version, tenant/session identifiers as appropriate). If you use data residency, the usage object also reports the inference_geo where inference actually ran — worth logging, since US-routed inference carries a 1.1x pricing multiplier.
import time, json
from anthropic import AnthropicAWS
client = AnthropicAWS()
t0 = time.monotonic()
resp = client.messages.create(model="claude-sonnet-5", max_tokens=1024,
messages=[{"role": "user", "content": question}])
print(json.dumps({
"event": "claude_call", "model": "claude-sonnet-5",
"latency_ms": int((time.monotonic() - t0) * 1000),
"input_tokens": resp.usage.input_tokens,
"output_tokens": resp.usage.output_tokens,
"stop_reason": resp.stop_reason,
"feature": "support-assistant", "prompt_v": "v14"}))The two request IDs arrive as response headers — capture them via your SDK's raw-response access or HTTP-layer logging and add them to the event. Logs go to stdout, CloudWatch Logs collects them, and CloudWatch Logs Insights (or your log platform of choice) answers ad-hoc questions like "which feature drove yesterday's output-token spike." Resist logging prompt or completion text into general application logs by default: it bloats storage, and it quietly turns your logging pipeline into a store of potentially sensitive data with its own retention obligations. If you need content for quality review, sample it deliberately into a separate, access-controlled store.
Metrics: publish the four that page people
From those same events, publish CloudWatch metrics — via the embedded metric format or explicit puts — dimensioned by model and feature: request latency (alarm on p95, and track time-to-first-token separately for streaming), error rate (split 429 throttling from 4xx configuration errors from 5xx — they have different owners and different fixes), token throughput (input and output tokens per minute; output is the early-warning signal for both cost and latency), and truncation/refusal rate (stop_reason of max_tokens or refusal as a fraction of calls — a quality signal that needs no human grader). CloudTrail complements this at the audit layer: workspace-level actions are management events logged by default, while inference calls are data events you must explicitly enable.
One dashboard, cost and quality together
The classic failure is a cost dashboard owned by finance and a latency dashboard owned by SRE, with nobody watching the interaction. Put them on one screen: estimated spend per feature (token counts × list prices — for example $3/$15 per million input/output tokens for Claude Sonnet 5 at standard pricing, $5/$25 for Opus 4.8), tokens per request trend, cache hit ratio if you use prompt caching (cache reads bill at roughly a tenth of base input price, so a falling hit ratio is a cost regression), alongside p95 latency, error rate, and truncation/refusal rate. The pattern to watch for: tokens-per-request drifting up over weeks — usually growing conversation history or prompt bloat — which degrades cost and latency together long before any single alarm fires.
Where to go next
For the audit-trail layer beneath these metrics, see CloudTrail logging and compliance patterns. Cost allocation goes deeper on attributing spend, and feature-flag rollout shows these signals driving ship/no-ship decisions.