LLM on

Inference disaggregation (part 2): moving the cache

Fri, 25 Sep 2026 11:10:00 -0700

Part 1 showed why you’d split prefill and decode onto different GPUs: a long prefill on the same GPU makes everyone who is already decoding there wait. Splitting has a price, though. The KV cache that prefill builds has to move to the GPU that does decode. The offload post (part 3 of Watching a KV cache grow) predicted that moving a 32k-token cache would add about 3% to prefill over a 400 Gb/s link and about 12% over 100 Gb/s. This post measures the fastest case, within one machine, on real hardware.

Inference disaggregation (part 1): why split prefill and decode

Fri, 25 Sep 2026 11:05:00 -0700

In Watching a KV cache grow I looked at the two phases of LLM inference: prefill, which processes a whole prompt in one go, and decode, which produces one token at a time. They behave very differently. Prefill is one big burst of work that scales with the prompt. Decode is a long run of small steps, each one reading the whole KV cache.

Disaggregated inference splits the work of serving a model across separate hardware. The most common form, and the one this series starts with, runs prefill and decode on different GPUs. Other forms split each layer’s attention and feed-forward parts onto different chips, or keep the KV cache in its own memory pool shared by many GPUs.

Watching a KV cache grow (part 3): move it or rebuild it?

Thu, 24 Sep 2026 15:54:50 -0700

Part 1 showed what a KV cache is as bytes: 23,040 per token for SmolLM2-135M. Part 2 showed that decode gets slower as the cache grows, because every step has to move it through memory.

Where I’m heading with this series is disaggregated inference. The idea is to run prefill on one set of GPUs and decode on another, because the two phases want different things (parts 1 and 2 are really about that difference). The catch is that the cache prefill builds has to get from the prefill machine to the decode machine before decoding can start.

Watching a KV cache grow (part 2)

Thu, 24 Sep 2026 12:20:56 -0700

In part 1 I watched a KV cache fill up one token at a time on my laptop, and worked out that SmolLM2-135M’s cache costs 23,040 bytes per token. The post ended with a question: every decode step has to read the weights and the whole cache, so as the cache grows, does each step get slower in proportion to the extra bytes?

The short answer is yes, in a straight line, but each step moves the cache through memory about 3 times rather than once. With a different cache class it’s closer to 7.

Watching a KV cache grow (part 1)

Thu, 24 Sep 2026 11:12:06 -0700

Glenn Lockwood’s What are KV caches, really? explains prefill, decode, and why a KV cache exists, and then takes apart some vendor speedup claims for KV offload. It’s a good read and it got me curious.

I’m an infra person, not an ML person, so I wanted to see a KV cache for real, the way I’d look at any other buffer: what’s in it, how it grows, and how big it gets.