Ask Claude "what is our parental-leave policy?" and, with no other help, it can only guess or decline — your HR handbook was never in its training data. Retrieval-augmented generation (RAG) solves this with an idea any librarian would recognize: before asking the question, fetch the relevant pages and hand them over with it. The model answers from the documents in front of it rather than from memory. That is the entire trick. Everything else — embeddings, vector databases, chunking — is plumbing in service of one step: find the right passages and put them in the prompt.
How a minimal RAG pipeline works
A typical pipeline has two halves. The first runs ahead of time: split your documents into passages ("chunks," often a few paragraphs each), compute a numerical fingerprint of each one's meaning (an "embedding"), and store the fingerprints in a searchable index — a vector database or the vector feature of a database you already run. The second half runs per question: embed the user's question the same way, find the passages whose fingerprints sit closest to it, and build a prompt that says, in effect, "Here are five excerpts from our documentation. Using only these, answer the following question. Cite which excerpt supports each claim."
Claude's role is the last step: reading the retrieved passages and writing a grounded answer. This works with the core Messages API, which is available on all four platforms — Bedrock, Vertex AI, Foundry, and Claude Platform on AWS — so RAG itself has no platform availability caveats. The retrieval infrastructure is yours and typically comes from your cloud's native search or database services.
A concrete example
An internal HR assistant is the classic starter project. You index the employee handbook, benefits documents, and IT policies — perhaps a few hundred pages. An employee asks, "Can I carry unused vacation into next year?" The system retrieves the three passages about leave carryover, and Claude answers in two sentences, quoting the policy and naming the document it came from. When the handbook changes, you re-index the changed pages; there is no retraining, because nothing was ever trained. That last property is why RAG beats fine-tuning for company knowledge: the knowledge lives in documents you can update, audit, and delete, not inside model weights.
Do you even need retrieval?
Sometimes not. Claude Opus 4.8 and Sonnet 5 accept up to one million tokens of context — on the order of thousands of pages — and Haiku 4.5 accepts 200K. If your corpus is one policy manual or a single contract bundle, you can skip the pipeline and include the whole thing in the prompt, using prompt caching (available on all four platforms) to avoid re-paying for those tokens on every request. RAG earns its complexity when the corpus is too large to send, changes frequently, or must be filtered per user.
The pitfalls that actually hurt
Retrieval quality is the ceiling. If the right passage is not retrieved, no model can answer correctly. Most "the AI is wrong" complaints in RAG systems are really "the search step returned the wrong pages." Invest in testing retrieval separately from generation.
Stale indexes. An assistant confidently quoting last year's travel policy is worse than no assistant. Re-indexing must be an automated part of your document workflow, not a manual chore.
Ignoring access control. The index does not know your permission model unless you build it in. Filter retrieval by the requesting user's entitlements; otherwise the assistant becomes a way to read documents someone was never allowed to open.
No citations. Require the prompt to demand quoted sources, and show them in the UI. Citations turn "trust the bot" into "check the reference," which is what makes review practical.
Answering beyond the documents. Instruct Claude to say "I don't find this in the provided documents" rather than fall back on general knowledge. A grounded "I don't know" is a feature.
Where to go next
See the internal knowledge assistant article for a full reference design built on RAG, and making sense of 1M-token context windows for when long context can replace retrieval entirely. The feature matrix covers what each platform supports.