Every retrieval-augmented generation (RAG) deployment eventually hits the same operational question: the source documents changed on Tuesday — when does the assistant stop citing Monday's version? Full re-ingestion answers it expensively and slowly. Incremental sync answers it cheaply, but only if you design for invalidation from the start. The core insight: your unit of invalidation should be the chunk, not the document, and the machinery that decides what changed should be deterministic code, with Claude involved only where enrichment adds value.
Detect changes deterministically
Give every source document a content hash and every chunk a hash of its own text plus its position. On each sync run: pull changed documents (most sources expose a modified-since API or webhook), re-chunk them with the same chunking rules as the original ingestion, and compare chunk hashes. Unchanged hashes keep their existing embeddings and enrichments; changed or new chunks go to the processing queue; hashes that disappeared get tombstoned in the index. This is pure code — no model calls — and it typically shrinks the work of a sync run to a small fraction of the corpus. Chunking rules are part of the contract: change them and every hash changes, which is a full re-index. Version the chunker and treat a version bump as a planned migration (see RAG chunking strategies).
Enrich only the changed chunks
Where teams use Claude in the ingestion path — generating chunk summaries, titles, or query-style paraphrases that improve retrieval — the incremental design pays off directly, because only dirty chunks incur model cost. This work is a natural fit for batch processing: Anthropic's Message Batches API accepts up to 100,000 requests per batch at 50% of standard token prices, each tagged with a custom_id (1–64 characters), and most batches finish in under an hour. Use the chunk hash as the custom_id so results map back to index rows without bookkeeping. Since batches can take longer than five minutes, put any shared enrichment instructions behind a 1-hour prompt-cache TTL rather than the default 5-minute one. One important caveat: batches expire if not finished within 24 hours, and results are downloadable for 29 days — build your sync job to collect results promptly, not to treat the batch store as an archive.
Storing source files with the Files API
If your pipeline keeps canonical copies of source documents for later question-answering, the Files API offers workspace-scoped storage (up to 500 MB per file, 500 GB per organization) that Messages requests can reference by file_id instead of re-uploading content. Its semantics actually suit sync work well: files cannot be modified or renamed after upload — the documented update path is "upload a new file, delete the old one" — which forces the same immutable-version discipline your chunk hashes already implement. Store the current file_id per document version in your sync metadata, and remember deletion is unrecoverable.
Freshness meets prompt caching
If your serving path caches a shared prefix (system prompt, tool definitions, or pinned reference documents), remember that caching is a strict prefix match — one changed byte invalidates every breakpoint after it. A knowledge refresh that edits pinned context will force cache rewrites at 1.25x the input price for the next request; that is expected and fine, but schedule bulk refreshes off-peak rather than mid-traffic. Chunks retrieved per-query sit after the cached prefix, so index updates do not disturb the cache at all — another argument for retrieving context dynamically instead of pinning it.
Platform availability
The deterministic machinery (hashing, diffing, queues, tombstones) is yours and runs anywhere. The Claude-dependent pieces vary: the Files API is available on the first-party Claude API and Claude Platform on AWS, in beta on Microsoft Foundry (Hosted-on-Anthropic deployments only), and not available on Amazon Bedrock or Google Cloud — on those platforms, keep canonical files in S3 or GCS and send content inline. Anthropic's Message Batches API is likewise absent on Bedrock and Vertex AI, but both clouds provide their own batch inference for Claude (S3-based on Bedrock; BigQuery/GCS-based on Vertex at 50% batch pricing), so incremental enrichment works everywhere with platform-appropriate plumbing. Prompt caching is available on all surfaces, with Bedrock supporting explicit breakpoints only.
Where to go next
Pair this with re-ranking for retrieval precision and the Files API article for storage mechanics. The feature matrix shows what runs where.