Microsoft Foundry's observability stack is genuinely broad: built-in and custom evaluators, Azure Monitor dashboards for token consumption, latency, error rates and quality scores, OpenTelemetry tracing, continuous evaluation of sampled production traffic, and explicit support for wiring evaluators into CI/CD pipelines as quality gates. Its evaluator catalog spans general-purpose quality (Coherence, Fluency), textual similarity (F1, BLEU, ROUGE, METEOR and friends), RAG metrics (Retrieval, Groundedness, Relevance), a long list of risk-and-safety checks, and agent evaluators. For a team standardizing on Azure, it is a serious evaluation platform.
Then comes the fine print that matters for Claude users.
The documented judge restriction
Many of Foundry's most useful evaluators are LLM-as-judge evaluators — a second model reads the response and assigns a score (typically on a 1–5 Likert scale, with a default pass threshold of 3). Microsoft's documentation is unambiguous about which models may sit in the judge's seat: "For LLM-as-judge evaluators, you can use Azure OpenAI or OpenAI reasoning and non-reasoning models for the LLM judge. For the best balance of performance and cost, use gpt-5-mini."
Claude is not on that list. Even though Claude models are deployable in Foundry — the Hosted-on-Azure variants of Opus 4.8, Sonnet 5, and Haiku 4.5 are generally available — Anthropic models are not documented as usable judges. Each judge evaluator takes a deployment_name pointing at your (OpenAI-family) judge deployment, incurs model inference costs per evaluation call, and for coherence/fluency currently supports English-language responses.
Can Claude deployments even be evaluation targets?
Here the documentation goes quiet rather than saying no. Microsoft's Claude models reference page — the one covering capabilities, quotas, and APIs — does not mention evaluators, evaluation runs, or the observability dashboards at all. Whether a Claude deployment can be the target of Foundry's built-in evaluation runs (the model whose endpoint the evaluation framework invokes) is simply not documented, in either direction. Resist the temptation to assume; if live-endpoint evaluation of a Claude deployment matters to you, verify with Microsoft against current documentation.
The static-data workaround
What clearly does work with any model's outputs is evaluating data rather than endpoints. Foundry's evaluators read their inputs through data mappings against your test dataset — fields like {{item.query}} and {{item.response}} — which makes the evaluation of an arbitrary dataset model-agnostic in practice. Generate responses with Claude yourself (through your Foundry deployment, or anywhere else Claude runs), write the query/response pairs into a dataset, and let Foundry's evaluators score them as static data. Two honest caveats: Microsoft does not state this pattern for Claude specifically, and the judging is still done by an Azure OpenAI/OpenAI model. Conceptually this mirrors Bedrock's bring-your-own-inference mode — decouple generation from grading, and the platform's judge restriction only constrains the grading half.
Choosing a path as a Claude-on-Foundry team
Three workable postures, in increasing order of effort. First, accept the cross-vendor judge: use Foundry's evaluators with an OpenAI-family judge over static Claude outputs, and get the CI/CD-gate and monitoring integration Foundry does well. Second, split the work: keep operational monitoring in Foundry, but run judge-based quality evaluation where Claude judges are documented — Bedrock, or a self-built harness following Anthropic's evaluation guidance (clear rubrics, forced score formats, reasoning before scoring). Third, go fully self-built for maximum control. Whichever you choose, note one adjacent Foundry fact: Microsoft states that "Foundry doesn't provide built-in content filtering for Claude models at deployment time," so safety evaluation and inference-time content safety configuration deserve their own line on your checklist, and Microsoft points to Anthropic's system cards for model-level safety evaluations.
Where to go next
See the Foundry deep dive for the platform fundamentals, and judge-model independence for how much the judge's vendor should worry you. The platform overview compares evaluation stories across all four Claude 3P routes.