Multimodal Turns: Mixing Text, Images, and Documents in the Messages Array

Vision and document input are not separate APIs — they are just more block types in the same messages array you already use for text. A single user turn can mix any number of text, image, and document blocks. What differs is how you get the bytes to the model, how much each modality costs in tokens, and which delivery methods your platform supports.

Three ways to deliver an image or PDF

Each image or document block has a source object, and there are three source types. Base64 embeds the encoded bytes directly in the request — universally supported and simplest, but it bloats payloads. URL ("source": {"type": "url", ...}) points the API at a publicly fetchable location. File references point at content you previously uploaded through the Files API with "source": {"type": "file", "file_id": "..."} — ideal when the same document is used across many requests, and the way to stay under the 32 MB request size limit when sending many images.

Platform support is the first gotcha: Amazon Bedrock and Google Vertex AI currently support base64 only for both images and PDFs. URL and file sources work on the Claude API and Claude Platform on AWS; the Files API itself is beta, is also available on Microsoft Foundry for Hosted-on-Anthropic deployments, and is not available on Bedrock or Google Cloud at all.

from anthropic import AnthropicBedrockMantle
import base64

client = AnthropicBedrockMantle(aws_region="us-east-1")
img = base64.standard_b64encode(open("chart.png", "rb").read()).decode()
resp = client.messages.create(
    model="anthropic.claude-sonnet-5", max_tokens=1024,
    messages=[{"role": "user", "content": [
        {"type": "image", "source": {"type": "base64",
         "media_type": "image/png", "data": img}},
        {"type": "text", "text": "Summarize this chart in two sentences."},
    ]}])

What the model sees for each type

Images (JPEG, PNG, GIF, WebP; animations use only the first frame) are processed in 28×28-pixel patches, costing ceil(width/28) × ceil(height/28) visual tokens — a 1000×1000 image is about 1,296 tokens. Newer high-resolution models (including Opus 4.8 and Sonnet 5) accept a longer edge of up to 2576 px and up to 4,784 visual tokens per image, roughly 3× more than standard models, so downsample before sending if you don't need the fidelity. Claude does not read image metadata and cannot identify people in photos.

PDFs get a dual treatment: each page is converted to an image and its extracted text is provided alongside, so you pay both image tokens and typically 1,500–3,000 text tokens per page. That is what lets Claude read charts and layout, not just the text layer. Dense PDFs can fill the context window well before page limits, so split large documents.

Text blocks are just tokens. Placement matters, though: Anthropic recommends putting images and documents before the text that asks about them, and its long-context guidance puts queries at the end for measurably better results.

Limits worth writing down

One subtlety: requests with more than 20 image or document blocks get a stricter per-image dimension limit — keep each dimension at or under 2000 px, or stay at 20 blocks or fewer, or you'll see an invalid_request_error mentioning "many-image requests."

Where to go next

The underlying block grammar is covered in Content Blocks. Since repeated documents are expensive, pair multimodal turns with caching — see Cache Read vs Write Tokens — and check the feature matrix for platform coverage.

Sources

• Vision
• PDF support
• Files API
• Claude prompting best practices