Six ways LLM serving got faster. Which ones do you control?

Prefix caching, continuous batching, PagedAttention, chunked prefill, speculative decoding, and split prefill and decode pools are why one GPU now serves more traffic. Some of them you control. Most of them you only buy. Here is which is which, and what to ask.

PBy Prestanda Consulting
Field Note · 13 Nov 2026

Blocks: prompt tokens. Thin plates: generated tokensPress: does the workShelf: the KV cacheThe press reuses cached blocks up to the first one that differs, then recomputes the rest. With the timestamp last, only it is recomputed. Put it first and nothing after it matches. vLLM, for example, caches full blocks only.

The short version

Six techniques do most of the work in a modern serving engine: prefix caching, continuous batching, PagedAttention, chunked prefill, speculative decoding, and splitting prefill from decode. Together they decide how much traffic a GPU serves and how quickly the first and next tokens arrive.

Who controls them depends on how you buy. Through an API, the provider decides which of them run. Prompt caching is the exception: you shape it with how you order your prompts, and it has its own line on the invoice. Run your own models and every one of the six is a setting your platform team owns.

Running the engine is the provider’s job. Prompt order, cache settings, and the latency target are yours.

The six, and who controls each

Technique	What it saves	Self-hosted (vLLM)	Through an API
Prefix caching	Prompt processing (prefill) for a prompt start the engine has already seen. Decoding still runs.	Caches full blocks only and evicts the least recently used first. `cache_salt` keeps tenants apart (vLLM, vendor).	On by default at OpenAI; opt-in cache checkpoints on Amazon Bedrock. Priced separately on both.
Continuous batching	GPU slots left idle while the longest request in a batch finishes.	The scheduler adds and removes requests at every step, the idea introduced as iteration-level scheduling in Orca (OSDI 2022, independent).	The provider’s choice. Not visible to you.
PagedAttention	GPU memory reserved for each request’s maximum length and never used.	Stores the KV cache in fixed-size blocks behind a block table, like operating-system paging (Kwon et al., 2023, independent).	The provider’s choice.
Chunked prefill	Running requests stalling behind one long prompt.	On by default in vLLM V1 where possible. `max_num_batched_tokens` trades time to first token against inter-token latency (vLLM, vendor).	The provider’s choice.
Speculative decoding	Time per output token, mainly at low to medium load.	EAGLE, multi-token prediction, draft models, n-gram, and suffix decoding. Lossless with the standard rejection sampler (vLLM, vendor).	The provider’s choice.
Prefill and decode on separate pools	Tail inter-token latency when prefills land in the middle of decoding.	Experimental. vLLM states it “DOES NOT improve throughput” (vLLM, vendor).	The provider’s choice.

Through an API: prompt caching is the one you tune

OpenAI. Caching is on by default for supported models. From GPT-5.6, a prompt needs at least 1,024 visible input tokens to be cached. Cache reads cost 0.1× the ordinary input rate (0.05× on GPT-6.1 Sol and Terra). Cache writes cost 1.25× with the default 30-minute retention and 2.5× with 12 hours. The whole prefix must match, and caches are not shared across organisations (OpenAI, vendor).

Amazon Bedrock. You mark cache checkpoints yourself: cachePoint in the Converse API, cache_control for Claude through InvokeModel. The default time to live is 5 minutes, with 1 hour on some models. Minimums run from 512 to 4,096 tokens per checkpoint, depending on the model. Caching works on on-demand endpoints only, not the batch inference API (AWS, vendor).

The consequence: on GPT-5.6 and later, a cache write costs more than ordinary input. A cache nobody reads again is extra cost.

The checklist

Step 1. Measure your cache-hit rate. Compare cached_tokens (OpenAI) or cacheReadInputTokens (Bedrock) with total input tokens, per workload. Owner: platform team.
Step 2. Put stable content first. Tools, system instructions, and shared reference material go first. Timestamps, user names, and retrieved records go last. One changed character early in the prompt breaks the match for everything after it. Owner: application team.
Step 3. Set the cache lifetime from your traffic. If requests that share a prefix arrive further apart than the time to live, you pay for writes and get no reads. Owner: platform team.
Step 4. Name the latency that matters. A chat assistant needs fast first tokens. A long report needs steady token flow. That one decision sets your chunk size (max_num_batched_tokens; vLLM gives 2,048 as an example of a value that favours inter-token latency) and whether separate prefill and decode pools are worth a test. Owner: product owner, with the platform team.
Step 5. Test speculative decoding on your own traffic. Gains depend on model, traffic, hardware, and sampling settings. vLLM reports acceptance metrics per request. Keep the standard rejection sampler unless you accept changed outputs. Owner: ML engineering.
Step 6. Keep tenants apart. On a shared self-hosted engine, give each tenant its own cache_salt. Without it, vLLM warns, response latency can reveal what another user has cached. Owner: security lead.

Where the bill moves

A faster engine is half of it. How you buy capacity is the other half. At QCon AI, Walmart Global Tech’s Aditya Mulik described a nightly multi-agent batch job that had 400 reserved provisioned throughput units, which he estimated at around $100k to $120k a month. After it moved to pay-as-you-go, he put the cost at about $10k a month, retries included (InfoQ, vendor: the speaker’s estimate). Reserved capacity pays when it is busy. A nightly job is not busy for most of the day.

Where this breaks

No cache hit is guaranteed. OpenAI caches live on individual machines, and traffic above about 15 requests per minute for one prefix can overflow to other machines. Bedrock says an eligible request is not guaranteed a hit.
Outputs can still differ between runs. vLLM does not guarantee stable logprobs. Batch size and floating-point precision can change outputs with and without speculative decoding. Its relaxed verification modes are not distribution-preserving.
Separate pools are not a throughput play. They add a KV-cache transfer between pools and exist to control latency.
Published speed-ups are not your speed-ups. The papers above measured their own models and hardware. Measure on yours.

Missing

A Prestanda measurement. We have not benchmarked any of the six for this note.
Caching terms for Azure OpenAI, Google Vertex AI, and Anthropic’s own API. We did not read them for this note.
SGLang and TensorRT-LLM. The self-hosted column reflects vLLM’s documentation only.
Whether vLLM turns prefix caching on by default. Its feature page was unreadable from our environment.

Frequently asked questions

Does prompt caching change the answer?

Caching reuses work already done for an identical prompt start. It skips prompt processing, not generation: the model still decodes every output token.

Is speculative decoding safe where outputs are audited?

vLLM says the standard rejection sampler is lossless up to the limits of hardware numerics, and its relaxed modes are not. It also does not guarantee stable logprobs between runs. Record which method is switched on.

Should we split prefill and decode onto separate GPUs?

Only if you have a strict latency target. vLLM marks the feature experimental and says it does not improve throughput. Its purpose is to keep long prompts from slowing tokens already being generated.

Living page. Last updated 13 Nov 2026. We update this note when providers change caching terms or when we run our own measurements. Want your prompts and capacity checked against your traffic? Talk to us.