Teams building document-heavy workloads on Google Vertex AI run into two distinct limits that are easy to conflate. One is a bytes limit: how big the HTTP request payload can be. The other is a tokens limit: how much material the model can hold in its context window. You can violate either one independently, and the error you get differs.
The two ceilings
| Ceiling | Value | What it governs |
|---|---|---|
| Request payload | 30 MB | Total bytes of the JSON request, including base64-encoded images (max 5 MB each, 100 per request) |
| Context window | 1M tokens (Fable 5, Opus 4.8/4.7/4.6, Sonnet 5, Sonnet 4.6); 200k for others incl. Haiku 4.5 and Sonnet 4.5 | Everything the model processes: system prompt, all messages, tool definitions, and the generated output |
Plain text rarely hits the 30 MB byte ceiling before hitting token limits — a million tokens of English is on the order of a few megabytes. The byte ceiling bites when payloads carry base64 images or PDFs, since base64 inflates binary data by roughly a third. Conversely, exceeding the token window with input alone returns a 400 invalid_request_error ("prompt is too long") — a validation failure, not a rate limit, so retrying will not help.
Chunking strategies for oversized inputs
When the corpus genuinely exceeds what one request can carry, the standard playbook applies:
Map-reduce summarization. Split the document set into chunks that each fit comfortably, ask Claude to summarize or extract from each chunk, then run a final pass over the intermediate outputs. Splitting on natural boundaries — sections, contracts, tickets — beats splitting on raw size, because each chunk stays self-interpreting.
Retrieval instead of stuffing. If the workload only ever needs a slice of the corpus per question, index the corpus and retrieve the relevant passages per request rather than shipping everything every time. This is usually cheaper even when the full corpus would fit.
Count before you send. Token counting is supported for Claude on Vertex, so a pre-flight count can route a request to the chunking path before it fails. Remember that with prompt caching, input_tokens, cache_read_input_tokens, and cache_creation_input_tokens all still occupy the window — caching changes what you pay, not whether tokens count.
Where Cloud Storage fits — and where it doesn't
A tempting idea is to sidestep payload bloat by passing gs:// URIs and letting the platform pull the files. For Claude's online requests on Vertex, that is not supported: Anthropic's documentation lists URL input sources and the Files API as unavailable on this platform. Your application must fetch the bytes and inline them.
Cloud Storage does solve a real payload problem in one place: Google's native batch prediction for Claude accepts input as JSONL files in Cloud Storage or as a BigQuery table, runs asynchronously (default four concurrent batch requests per project; the global endpoint is not supported for batch), and is billed at 50% of on-demand rates. For "process this archive of 100,000 documents" workloads, that is the right tool — not a bigger online request.
Mind the long-context meter
Google's pricing page for Claude states that if a query's input context is 200K tokens or longer, all tokens in the request — input and output — are charged at long-context rates, listed in per-model ">200K" columns. So on Vertex, crossing the 200K threshold changes the unit price of the whole request, not just the excess. Before standardizing on giant single requests, price a representative request at your actual sizes against the current pricing page; chunked processing under the threshold is sometimes cheaper than one heroic call.
Where to go next
Long requests should almost always stream — see Streaming Claude Responses on Vertex AI — and repeated long prefixes should be cached — see Prompt Caching on Vertex AI.