AI systems / inference serving

LLM serving: why prefill and decode want different machines

An LLM request has two very different computational phases: process the existing prompt, then generate one token at a time. Treating them as one batch is simple, but the phases fight over GPU compute, memory bandwidth, and latency. The key to separating them is the key/value cache: a large, structured tensor that must move from the prefill worker to the decode worker without changing its meaning.

AI systems deep dive · KV-cache movement and disaggregated inference

2026-08-19

The concrete problem

Suppose a user sends an 8,000-token prompt and expects the first answer token quickly. The server must run the transformer over all 8,000 input positions before it can generate token 8,001. That initial pass is called prefill. Afterward, each new token depends on all previous tokens, but only one new position is computed per step. That loop is called decode.

Prefill wants large matrix multiplications and enough parallel work to keep tensor cores busy. Decode has a tiny amount of new arithmetic but repeatedly reads the accumulated attention state from memory. Prefill is often compute-bound; decode is commonly memory-bandwidth- and latency-bound. A long prompt arriving at the wrong moment can therefore make existing chat sessions miss their per-token latency target even though the GPU has plenty of arithmetic capacity left during decode.

What the key/value cache contains

At each transformer layer, self-attention projects hidden states into queries, keys, and values. A new decode token needs a query, but it must compare that query with the keys and values for every earlier token. Recomputing those earlier projections would make generation needlessly expensive, so the server stores them in the KV cache.

For a model with L layers, HKV key/value heads, head dimension D, and B bytes per scalar, the approximate cache bytes per token are:

bytes_per_token ≈ 2 × L × H_KV × D × B

The factor 2 is for K and V. Grouped-query attention reduces HKV below the number of query heads, which is why two models with similar parameter counts can have very different decode memory requirements.

For L=32, HKV=8, D=128, and BF16 (2 bytes), one token costs about 128 KiB. An 8,192-token prompt then contributes about 1 GiB of K/V state for one sequence. That is the handoff payload before transport headers, staging buffers, alignment, replication, or a second copy on the receiving device.

The cache is not necessarily stored as one giant contiguous tensor. Variable-length requests arrive and finish continuously, and a contiguous allocation would force expensive copies or leave unusable holes. A paged design divides the sequence into fixed-size token blocks. The logical sequence has a block table; the physical blocks can live anywhere in a pool.

Why one scheduler becomes two

Early high-performance serving systems improved throughput by scheduling at iteration granularity rather than waiting for an entire batch to finish. That lets a newly arrived request join a running batch and lets finished sequences leave. PagedAttention then made the cache itself shareable and fragment-resistant. Those ideas remain useful even when prefill and decode are split.

A single worker can still mix both phases, but it has to choose a policy. If it admits a large prefill, decode iterations wait behind a compute-heavy operation. If it protects decode, prompt processing is delayed and TTFT rises. Dynamic batching can reduce idle time, yet it cannot remove the physical interference between a long matrix workload and bandwidth-sensitive decode.

Disaggregated serving creates separate pools:

There is a practical asymmetry: prefill can often be batched across many prompt tokens, while decode is a repeated synchronization point. A decode worker may hold dozens of active sequences, but each iteration still needs every participating sequence’s current block table and output. This makes queueing policy and straggler handling part of model performance, not merely infrastructure plumbing.

The open-source implementation shape

Names change between releases, but a modern engine has roughly these objects and paths:

• Request and sequence state: token IDs, sampling parameters, finished status, prefix-cache identity, and the number of computed tokens.
• Block manager: maps logical token ranges to physical KV blocks, maintains free blocks and reference counts, and updates the sequence’s block table as decode appends tokens.
• Scheduler: decides which requests receive the next iteration budget, whether a prompt is prefilling, and whether a decode sequence may advance.
• Attention layer and kernel: receives query tensors plus a KV-cache handle. A paged kernel uses block tables and slot mappings to gather the right keys and values instead of assuming contiguous history.
• KV connector: exports or imports cache blocks, carries tensor layout metadata, and reports transfer completion or failure to the scheduler.

In vLLM, useful source landmarks include vllm/attention/layer.py, the v1 scheduler and GPU model runner, and the paged-attention CUDA kernels under csrc/attention/. The exact paths are release-sensitive, so search the checkout for block_table, slot_mapping, num_computed_tokens, and kv_cache. The important interface is not a class name: it is the contract that maps a logical token position to a physical cache slot and makes that mapping available to the kernel.

A transfer-aware control flow looks like this:

Step 6 must be atomic from the decode scheduler’s point of view. Installing half a block table creates a particularly nasty failure: attention may read a valid-looking address containing another request’s data. The safe failure is to keep the request out of the runnable set until every required block is validated.

How much transfer is too much?

Let K be the prompt-cache bytes, R the effective end-to-end transfer rate, and Q the queueing and coordination delay. The minimum handoff contribution to TTFT is roughly:

handoff_time ≥ K / R + Q

For the example 1 GiB cache, an ideal 200 GiB/s path takes about 5 ms. A PCIe or RDMA path may have a much lower effective rate once staging, registration, copies, topology, flow control, and concurrent traffic are included. If the request will generate hundreds of tokens, that handoff may be worthwhile because it protects decode capacity. If the request is cancelled after its first token, the transfer was pure waste.

Some systems therefore use a threshold: short prompts stay on one worker, while long prompts or latency-sensitive traffic use the split path. Another option is layerwise or streamed transfer, allowing decode to begin when the first required layers arrive. That reduces startup latency but complicates scheduling, because a decode kernel must not touch a layer whose K/V is still in flight. It also increases the number of synchronization points and makes partial-transfer cancellation more expensive.

The current hardware conversation makes this division concrete. A recent report on Cerebras CS-4 describes prompt processing being offloaded to other accelerators while the wafer-scale system emphasizes decode, high on-device SRAM bandwidth, and low-latency interconnect. The durable lesson is architectural: the best device for building the cache need not be the best device for repeatedly reading it.

Failure modes and security boundaries

• Wrong model revision: a cache produced with different weights, rotary-embedding parameters, tokenizer rules, or tensor parallel layout is not reusable. Include a model and layout fingerprint in the transfer envelope.
• Block-table corruption: an off-by-one token offset or a stale physical block can produce plausible but incorrect text rather than a crash. Validate ranges, generation IDs, and checksums; keep block ownership explicit.
• Fragmentation: fixed pages reduce large-copy pressure but the final page of each sequence wastes capacity. Track allocated tokens versus live tokens and tune page size against fragmentation and kernel efficiency.
• Head-of-line blocking: a large transfer can occupy the same fabric as decode collectives. Use separate lanes or priorities, and measure transfer queue time separately from copy time.
• Cancellation leaks: if a client disconnects during handoff, both workers must release reservations. A leaked cache block is invisible model memory pressure until the allocator fails minutes later.
• Tail amplification: the decode pool may wait for the slowest shard or the slowest transfer. Set deadlines, make the request retryable only before ownership changes, and avoid duplicating a partially installed cache.
• Cross-tenant disclosure: K/V bytes encode the prompt and generated context. Treat them as sensitive user data. Zero or quarantine recycled blocks where required by the threat model, authenticate transfer endpoints, encrypt RDMA or network paths, and never put raw cache contents in crash dumps or debug logs.
• DMA and privilege: peer-to-peer GPU or RDMA access turns memory registration into a capability. Restrict which workers can register buffers, isolate tenants at the device/IOMMU boundary, and validate the request ID independently of an untrusted block ID.
• Side channels: cache occupancy, transfer size, and timing can reveal prompt length or sharing. Apply admission quotas and avoid exposing per-request memory statistics to tenants.

Debugging method

1. Split TTFT into queue wait, prefill compute, block export, transfer queue, copy, validation, and decode admission. One “TTFT” histogram hides the actual bottleneck.
2. For each handoff, log only identifiers and sizes: request ID, model fingerprint, number of blocks, logical token range, bytes, source/destination worker, start/end timestamps, checksum result, and release timestamp. Do not log K/V payloads.
3. Compare live_tokens, allocated_slots, and free_blocks. A growing difference indicates fragmentation or a release path bug; a falling free-block count with stable traffic suggests a leak.
4. When output is wrong, first force the same worker for prefill and decode. If the error disappears, compare dtype, tensor-parallel rank order, RoPE position offsets, block-table ordering, and model fingerprints before investigating sampling.
5. When ITL spikes, correlate GPU memory bandwidth, decode batch size, transfer bytes, NCCL/RDMA counters, and scheduler runnable counts. A high GPU utilization percentage can coexist with poor token latency if the utilization is prefill work.
6. Test cancellation, retry, worker death, and a full cache pool under load. These paths exercise ownership transitions that a happy-path benchmark never reaches.

Compact reproducible lab

Goal

Calculate the KV-cache payload for a real-sized configuration, estimate ideal handoff time, and see why paged allocation trades external fragmentation for bounded internal waste. This is a model of the memory contract, not a claim that a laptop can reproduce GPU kernel performance.

Prerequisites

Python 3.9 or newer. The lab uses only the standard library and runs on macOS, Linux, or Windows.

Exact commands

python3 - <<'PY'
from math import ceil

def kv_bytes(tokens, layers=32, kv_heads=8, head_dim=128, bytes_per_value=2):
    return tokens * 2 * layers * kv_heads * head_dim * bytes_per_value

def mib(n):
    return n / (1024 ** 2)

layers, kv_heads, head_dim, dtype_bytes = 32, 8, 128, 2
for prompt_tokens in (512, 8192, 32768):
    payload = kv_bytes(prompt_tokens, layers, kv_heads, head_dim, dtype_bytes)
    print(f"{prompt_tokens:5d} tokens: {mib(payload):8.1f} MiB")

payload = kv_bytes(8192, layers, kv_heads, head_dim, dtype_bytes)
print(f"ideal 200 GiB/s handoff: {payload / (200 * 1024**3) * 1000:.2f} ms")

for block_tokens in (16, 32, 64):
    sequences = (100, 37, 257)
    allocated = sum(ceil(n / block_tokens) * block_tokens for n in sequences)
    waste = allocated - sum(sequences)
    print(f"block={block_tokens:2d}: allocated={allocated:3d}, "
          f"waste={waste:2d} tokens ({waste / allocated:.1%})")
PY

Expected result

The 8,192-token prompt should be about 1024.0 MiB, and the idealized 200 GiB/s transfer should be about 5.00 ms. The page-size lines show the tradeoff: larger pages need fewer block-table entries but can waste more capacity in partially filled final pages. Real systems also pay alignment, metadata, transport, and concurrency costs.

Revealing variations

• Change kv_heads from 8 to 32. This approximates moving from grouped-query attention toward one K/V head per query head and shows why the same context length can require four times the cache.
• Change the transfer rate from 200 to 25 GiB/s. The payload is unchanged, but the split design now adds roughly 40 ms of ideal copy time before queueing and validation.
• Replace sequences with many short requests, such as (3,) * 100. The output makes internal page waste visible; this is why page size is a workload and kernel-design parameter, not a universal constant.

Primary sources

• Kwon et al., vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention — the paged KV-cache design and serving results.
• Yu et al., Orca: A Distributed Serving System for Transformer-Based Generative Models — iteration-level scheduling and why request-level batching is insufficient.
• Qin et al., DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving — the prefill/decode split and placement tradeoffs.
• vLLM attention kernels — block-table-driven GPU attention implementation; paths can move between releases.
• vLLM source tree — scheduler, model runner, KV-cache, and connector implementations.
• Agrawal et al., Sarathi-Serve — chunked prefill as an alternative way to reduce prefill/decode interference.
• The Register, Cerebras CS-4 report, 2026-08-19 — current hardware context for decode-focused, disaggregated inference.