If you run Claude on Google Cloud, the natural question is whether Vertex AI's built-in evaluation tooling — the Gen AI evaluation service — can score it, or whether you need to build your own harness. The short answer, as of July 2026: Google documents partner-model evaluation support that covers Anthropic models, but the Claude-specific details are thinner than on Bedrock, and at least one important question (Claude as a judge) is not documented at all. This article separates the documented from the undocumented, because the difference matters when you're planning an evaluation program.
What the service offers
The Gen AI evaluation service supports three metric families, per Google's documentation on choosing an evaluation approach:
Computation-based metrics — deterministic algorithms such as ROUGE, BLEU, and exact match, scored against reference answers. These need no judge model and are the most model-agnostic part of the service.
Model-based metrics — pointwise (score one response) and pairwise (pick the better of two responses) grading performed by a judge model. Judge configuration has its own "Configure a judge model" documentation page.
Managed rubric-based metrics — evaluation against structured rubrics managed by the service. Agent evaluation is documented as well, and the service is reachable through several interfaces: a console-based tutorial, the GenAI Client, and the evaluation module in the Agent Platform SDK.
What Google documents about partner models
Google's evaluation overview states that the service "supports evaluating partner models, such as Anthropic and Llama models," and that partner-model evaluation runs through Model Garden — "you must enable the model before running evaluations against a partner model." Enabling Claude in Model Garden is the same prerequisite as for any Claude use on Vertex (walkthrough in the Model Garden setup article).
On cost, Google's documented position is that third-party evaluation charges are based on the model inference incurred in Model Garden — in other words, evaluation runs against Claude generate ordinary Claude inference at Claude rates, rather than a separate flat evaluation fee for the model calls.
The documentation gaps
Two gaps deserve explicit flags, because on other platforms the answers are written down and here they are not.
Claude as judge: not documented. Google documents Claude's eligibility as an evaluated (partner) model. Whether Claude can serve as the judge/autorater in model-based metrics is not documented in the pages retrievable as of July 2026. Contrast this with Bedrock, which publishes an explicit Claude judge list (see the Bedrock judge article). If judge-model choice matters to your methodology — for example, you want a non-OpenAI, non-Gemini judge for independence reasons — verify directly with Google rather than assuming either way.
Per-feature Claude coverage: check page by page. Google organizes its evaluation docs into separate tracks — "Evaluating Gemini models," "Evaluate agents," "Evaluating a judge model," "Evaluating partner models" — and several adjacent Vertex features are Gemini-first. A capability documented on a Gemini-track page should not be assumed to extend to Claude; look for the partner-model track or an explicit statement.
A pragmatic approach for Claude-on-Vertex teams
Lean on the parts with the least ambiguity first. Computation-based metrics score text you supply against references you supply, which makes them the safest starting point regardless of how the partner-model story evolves. For judge-based scoring where you specifically want a Claude judge, remember you are not locked to one cloud's tooling: Bedrock accepts pre-generated responses from any source — including Claude on Vertex — via bring-your-own-inference, and Anthropic's own evaluation guidance supports building a lightweight LLM-graded harness yourself with whatever judge you choose. Treat Vertex's managed service as one option in the toolbox, sized to what its documentation actually promises today.
Where to go next
See the Vertex AI deep dive for the platform fundamentals, and judge-model independence for how to think about who grades whom. The platform overview compares all four Claude 3P routes.