Retrieval-augmented generation (RAG) is the pattern where you fetch relevant passages from your own documents and hand them to the model alongside the question. Before any retrieval happens, you must split documents into chunks — the units that get indexed, retrieved, and stuffed into the prompt. Chunking is unglamorous, invisible in demos, and the single most common root cause when a RAG system gives vague or wrong answers. The strategies below are established engineering practice; the Claude-specific facts about context, citations, and caching are drawn from the official docs.
The three strategies
| Strategy | How it splits | Wins when | Fails when |
|---|---|---|---|
| Fixed-size | Every N tokens, usually with overlap | Uniform content; you need something working today | Ideas straddle boundaries; answers arrive half-cut |
| Sentence/structure-aware | On sentence, paragraph, or heading boundaries up to a size cap | Well-formatted prose, manuals, policies | Tables, transcripts, poorly structured scans |
| Semantic | Where the topic shifts (detected by embeddings or a model pass) | Long mixed-topic documents; quality-critical corpora | Budget-sensitive ingestion — it costs a preprocessing pass per document |
The pragmatic recommendation: start structure-aware. It captures most of semantic chunking's quality at fixed-size cost, because human authors already put topic boundaries at headings and paragraphs. Reserve semantic chunking for corpora where structure-aware measurably fails, and keep fixed-size as the fallback for content with no usable structure. Whatever you pick, store each chunk with its source metadata — document ID, section title, page — because you will need it for citations and debugging.
Size chunks in tokens, and measure honestly
Chunk budgets are token budgets, and Claude's tokenizer is not OpenAI's — tiktoken undercounts Claude tokens by roughly 15–20% on typical text. Use the free count_tokens endpoint (available on the Claude API, Claude Platform on AWS, Bedrock, and Vertex AI; beta on Foundry) to measure real chunks from your corpus. Counts are also model-specific: Opus 4.7+, Fable 5, and Sonnet 5 use a newer tokenizer producing about 30% more tokens than earlier models for the same text, so recount if you switch models.
How Claude changes the classic trade-offs
Big context loosens the squeeze. Classic RAG advice assumed you could afford only a few small chunks. With 1M-token context windows on most current models (200K on Haiku 4.5), you can retrieve generously — larger chunks, more of them — and let the model sort relevance. Follow the documented long-context guidance: put retrieved content at the top of the prompt inside <document> tags with <source> subtags, and the user's question at the end; end-placed queries can improve response quality by up to 30%.
Citations reward one-chunk-per-document-block. Claude's citations feature ("citations": {"enabled": true} on document blocks) grounds answers in your chunks. The docs give explicit RAG guidance here: put each retrieved chunk in its own plain-text document block to get sentence-level citations, or use custom-content document blocks when you want citations to map exactly to your own units with no further chunking. Citations are GA on the Claude API, Claude Platform on AWS, Bedrock, Vertex AI, and Foundry — one of the more portable grounding features. Note the documented constraint that citations cannot be combined with structured outputs in the same request.
Caching punishes churn at the front of the prompt. Prompt caching is a strict prefix match: your system prompt and instructions cache cheaply (cache reads bill at 0.1x base input price), but the retrieved-chunk section changes per query and won't cache. Keep everything stable — instructions, format rules, few-shot examples — ahead of the volatile chunk block, and mind the per-model minimum cacheable length (1,024 tokens on Opus 4.8, Sonnet 5, and Haiku 4.5; 512 on Fable 5).
Where to go next
If you'd rather skip the vector store entirely, read citation grounding without embeddings; for the retrieval layer itself, see hybrid retrieval. The quickstart covers first calls per platform.