Retrieval-augmented generation (RAG) grounds answers in your documents. In the classic version your pipeline embeds the user's question, fetches the top-k chunks, and stuffs them into the prompt — one shot, fixed query. That fails on multi-part questions ("compare our 2024 and 2025 refund policies") and on questions phrased nothing like the source text. Agentic RAG fixes this by making retrieval a tool Claude calls itself, inside a loop your application runs.
The loop mechanics
Define a client tool — say search_kb — with a name, description, and JSON Schema input_schema. When Claude wants to search, the response comes back with stop_reason: "tool_use" and one or more tool_use blocks; your code runs the actual vector or keyword search and returns the chunks as tool_result blocks in the next user message. Claude may request several searches in one response (send all results back in a single message), and it keeps looping — refining queries, following leads — until it answers in plain text. Because tool use is a core Messages API capability, this loop runs on all four 3P platforms: Claude Platform on AWS, Amazon Bedrock, Google Vertex AI, and Microsoft Foundry.
Two small settings improve reliability: add strict: true to the tool definition so search arguments always match your schema, and return failed searches as a tool_result with "is_error": true and an informative message — Claude will rephrase and retry rather than hallucinate around a silent failure. Leave tool_choice at the default auto so Claude can answer directly when retrieval is unnecessary.
Self-querying and iterative refinement
The value of the agentic version is that query formulation becomes the model's job. Claude will decompose a comparison question into two searches, broaden a query that returned nothing, and issue follow-up searches when a chunk references another document. Encourage this in the system prompt: state that multiple searches are expected, that an empty result means "rephrase, don't give up," and — critically — that the answer must come only from retrieved content, with "not found in the knowledge base" as an acceptable answer. Put a loop cap in your application code (for example, stop accepting tool calls after N iterations and ask Claude to answer with what it has); deterministic guardrails belong in code, not prompts.
Source attribution with citations
Return retrieved chunks as document content blocks with "citations": {"enabled": true} rather than pasting them as plain text. Claude's answer then carries citation blocks with the exact cited_text and a document_index your UI can map back to the source file and location. For granularity control, put each chunk in its own plain-text document (you get sentence-level citations) or use custom content documents if your chunks should be cited as-is with no further splitting. A pleasant billing detail: cited_text does not count toward output tokens. One constraint: citations cannot be combined with structured outputs in the same request — if you need machine-readable answers and citations, split into two passes.
Keeping a long loop affordable
Every iteration resends the whole conversation, so costs compound. Three documented levers:
| Lever | What it does | 3P availability |
|---|---|---|
| Prompt caching | Cache system prompt + tool definitions; reads bill at 0.1x input price | All platforms (Bedrock: explicit breakpoints only) |
| Context editing | Server-side clearing of stale tool results past a token trigger | Beta on all five surfaces |
| Model tiering | Haiku 4.5 for routine lookups, Sonnet 5 / Opus 4.8 for synthesis | All platforms |
stop_reason and iteration counts from day one so you can tell which you have.Where to go next
Start with RAG basics if the classic pattern is new, then tool use 101 for loop plumbing and citations for attribution details.