chaowen. · All notes

2026-10-10 · AI/ML systems / inference memory

Technical deep dive

TurboQuant and the KV cache: compressing what inference remembers

Long-context language models spend memory not only on model weights, but also on the key/value (KV) cache: the attention data saved for every token so the model need not recompute its past at every generation step. Google Research’s TurboQuant proposal targets that cache with extremely low-bit vector quantization. The important lesson is not “compression is free”; it is how representation, correction, and measurement fit together.

Why the cache grows

For each layer and token, attention stores key and value vectors. A rough cache size is 2 × layers × tokens × KV heads × head dimension × bytes per element. The factor two is keys plus values. With long prompts, concurrent requests, or many layers, this memory can rival or exceed model-weight memory. Every byte removed can increase context length or the number of simultaneous requests that fit on a GPU.

prompt tokens → [K,V] per layer → retain for future attention
longer context = more retained vectors = more memory

Quantization is a geometry problem

A simple scalar quantizer rounds each coordinate to one of a small number of levels. But a vector’s useful information may be distributed across coordinates; rounding errors can align with directions that attention is sensitive to. Vector quantization instead represents a vector using a compact code, then reconstructs an approximation. TurboQuant combines a quantization scheme called PolarQuant with a small correction mechanism called Quantized Johnson–Lindenstrauss (QJL), intended to control distortion in the reconstructed vectors.

Conceptually, encode vector x as a code c, reconstruct x̂, and preserve a compact correction that helps reduce harmful error:

x  ──encode──> compact code c ──decode──> x̂
error e = x − x̂
objective: reduce memory while keeping attention behavior useful

Why error matters more than visual similarity

The cache is not an image to inspect. Its quality is judged downstream: does attention still retrieve the right information, and do generated answers remain useful? A tiny average reconstruction error can still damage rare but important directions. Therefore evaluation must include long-context retrieval and language-model task metrics, not just compression ratio or mean squared error.

Google’s March 2026 research announcement reports very large cache-memory reductions and benchmark results on selected models and tasks. Those are research results, not a blanket guarantee of identical quality for every model, workload, sequence length, or serving stack. Treat them as an invitation to reproduce the evaluation on your own distribution.

A reproducible toy lab

This small experiment shows the first-order memory arithmetic. It does not implement TurboQuant or validate model quality; it helps identify whether KV memory is even a meaningful bottleneck in your setup.

# Example: 32 layers, 8 KV heads, 128 dimensions, 8192 tokens
layers, heads, dim, tokens = 32, 8, 128, 8192
bytes_per_value = 2  # FP16
cache_bytes = 2 * layers * heads * dim * tokens * bytes_per_value
print(cache_bytes / 1024**3, "GiB")

# Compare a hypothetical 3-bit representation, ignoring metadata/overhead:
compressed = cache_bytes * (3 / 16)
print(compressed / 1024**3, "GiB (idealized)")

Run it in any Python 3 REPL. Then vary token count and KV-head count. The result is an idealized lower bound: real formats need scales, alignment, codebooks, kernels, and temporary buffers. A useful follow-up benchmark measures peak GPU memory, tokens per second, latency under concurrency, and task quality at the same context lengths.

Practical engineering checklist

Sources