RAG — retrieval-augmented generation — means fetching relevant documents from your own knowledge base and handing them to Claude alongside the user's question, so answers come from your content rather than the model's general training. Teams usually invest heavily in the retrieval side (embeddings, chunking, ranking) and then write a one-line prompt: "Answer the question using the context below." That line is where grounded systems quietly become ungrounded. The grounding prompt deserves the same engineering attention as the retriever, and everything in this article works identically whether you call Claude through Amazon Bedrock, Google Vertex AI, Microsoft Foundry, or Claude Platform on AWS — it is all standard Messages API prompting.
Structure the context so Claude can see its edges
Claude follows grounding instructions far more reliably when the retrieved material is clearly delimited. Anthropic's prompting guidance recommends wrapping each document in its own <document> tag, with <document_content> and <source> subtags, so instructions, context, and the question each live in named sections the model can parse. For long inputs — roughly 20,000 tokens and up — put the documents at the top of the prompt and the user's question at the end; Anthropic reports that placing queries after longform data can improve response quality by up to 30%.
Then make the boundary explicit in the instructions themselves. A workable grounding block looks like this:
Answer using ONLY the documents provided above.
- If the documents fully answer the question, answer and name the
source document(s) you used.
- If the documents partially answer it, answer the covered part and
state plainly what the documents do not cover.
- If the documents do not contain the answer, say: "I can't answer
that from the provided documents." Do not answer from general
knowledge. Do not guess.Say why, not just what
Anthropic's best-practices guidance emphasizes explaining the reason behind an instruction, because Claude generalizes from the explanation. "Only use the provided documents" is weaker than "Only use the provided documents — this answer will be shown to customers as official policy, so anything not in our documentation could expose us to incorrect commitments." The second version tells Claude how to handle the edge cases you didn't enumerate: adjacent-but-unstated facts, plausible inferences, outdated general knowledge about your company.
The insufficient-context branch is the product decision
What Claude says when retrieval comes back thin is a business choice, not a model behavior. Decide it deliberately: should the assistant refuse flatly, suggest a rephrasing, name the nearest topic it can answer, or hand off to a human? Write the exact fallback sentence into the prompt rather than leaving the wording to the model, so your support team and your logs see a consistent, searchable signal. Pair this with a retrieval-quality metric: if the fallback fires often, the fix is usually in the retriever, not the prompt.
Use citations to make grounding verifiable
Prompted grounding tells Claude to stay inside the context; the Messages API citations feature lets you verify that it did. Enable it per document with "citations": {"enabled": true} on document content blocks, and Claude's answer comes back with cited_text spans tied to a document_index. For RAG specifically, Anthropic's docs recommend putting each retrieved chunk in its own plain-text document (which yields sentence-level citations) or using custom content documents when you want your chunks used exactly as-is with no further splitting. Cited text does not count toward output tokens, so this is cheaper than prompting Claude to quote passages verbatim.
One constraint to plan around: citations cannot be combined with structured outputs — enabling both in one request returns a 400 error. If your pipeline needs machine-readable JSON and citations, split it into two steps.
Test grounding like a feature
Grounding failures are quiet: the answer sounds right and nobody checks the source. Build a small evaluation set with three kinds of questions — answers fully in the corpus, answers partially in the corpus, and answers definitely not in the corpus — and check that each triggers the correct branch. Re-run it whenever you change the prompt, the chunking, or the model. Anthropic's prompt-engineering guidance is blunt that success criteria and empirical tests are prerequisites, not afterthoughts.
Where to go next
For the retrieval side of the pipeline, start with RAG basics. Related prompting techniques are covered in reducing hallucination through prompt design and asking Claude to cite its sources.