Retrieval-augmented generation (RAG) pipelines usually retrieve with embeddings: cheap vector similarity that scores a query against every chunk independently. That first pass is fast but coarse — it ranks by topical closeness, not by whether a chunk actually answers the question. A re-ranker is a second, more expensive stage that reads the query and each candidate chunk together (the "cross-encoder" idea) and re-orders the top candidates before they reach Claude's context. The pattern: retrieve 50–100 candidates cheaply, re-rank them, keep the best handful.
When re-ranking closes the gap
Re-ranking pays off when your failure mode is "the right chunk was retrieved but ranked 14th, so it got cut." It does not fix a corpus that never retrieved the right chunk at all — that is a chunking or indexing problem (see RAG chunking strategies and hybrid retrieval). Run a quick audit first: for a sample of failed queries, check whether the correct chunk appears anywhere in the top 100. If yes, re-rank. If no, fix retrieval.
Using Claude as the re-ranker
You can use a dedicated re-ranker model, or use a small Claude model as an LLM judge. Claude Haiku 4.5 at $1 per million input tokens and $5 per million output tokens is the natural fit: re-ranking prompts are short, outputs are tiny, and latency matters. Ask for structured output so the score is machine-readable:
from anthropic import AnthropicVertex
client = AnthropicVertex(project_id="my-proj", region="global")
resp = client.messages.create(
model="claude-haiku-4-5",
max_tokens=200,
output_config={"format": {"type": "json_schema", "schema": {
"type": "object", "additionalProperties": False,
"required": ["relevance", "reason"],
"properties": {
"relevance": {"enum": ["high", "medium", "low", "none"]},
"reason": {"type": "string"}}}}},
messages=[{"role": "user", "content":
f"Query:\n{query}\n\nPassage:\n{chunk}\n\nRate relevance."}],
)One schema detail matters here: structured outputs do not support numeric constraints such as minimum/maximum (using them returns a 400 error), so you cannot enforce "a float between 0 and 1" at the schema level. An enum of discrete grades, as above, is more robust anyway — LLM judges are better at ordinal judgments than at calibrated decimals. Map grades to numbers in code (high=3 … none=0) if downstream ranking needs numerics.
Score normalization and assembly
Your first-stage retriever returns similarity scores on its own scale; your re-ranker returns grades on another. Do not average raw scores across stages — normalize in application code (rank-based fusion is a simple, robust choice) and treat the re-ranker as the tiebreaker among the retriever's top candidates. Deterministic transforms are code's job, not the model's. Since every candidate for one query shares the same query text and instructions, structure the prompt so the shared prefix comes first and mark it with a prompt-cache breakpoint; cache reads cost one-tenth of the base input price. For offline evaluation runs that re-rank thousands of stored queries, the Message Batches API halves the price again.
Feeding the winners to Claude
Once the top chunks are chosen, pass each one as its own plain-text document block with citations enabled — you get sentence-level citations back, so the final answer is grounded in the specific chunks that survived re-ranking. If you need coarser or finer granularity, custom content documents are used as-is with no further chunking.
Platform availability
The pieces this pattern uses travel well. Structured outputs and strict tool use are generally available on the Claude API, Claude Platform on AWS, Amazon Bedrock, and Vertex AI, and in beta on Microsoft Foundry. Citations and prompt caching follow the same shape (caching on Bedrock requires explicit breakpoints — no automatic caching). The one real gap is batch evaluation: Anthropic's Message Batches API is not available on Bedrock or Vertex AI, though both clouds offer their own cloud-native batch inference for Claude (S3-based on Bedrock; BigQuery/GCS-based on Vertex at 50% batch pricing). Your embedding store and first-stage retriever sit outside Claude entirely, so they are platform-neutral by construction.
Where to go next
Pair this with citation grounding for verifiable answers, and see Haiku routing for the cost logic of using a small model in the hot path. The feature matrix summarizes per-platform support.