llm-d: How KV Cache Became Cluster-Wide State (#AI)
Architectural ideas can have two birthdays: first, a paper shows that a technique works; later, someone turns it into a system that can be deployed and operated. This PyTorch Conference 2025 talk by Google’s Cong Liu and Maroon Ayoub, then at IBM Research, is about that second birthday. llm-d did not invent Prefill/Decode disaggregation. It is attempting to turn a research pattern into an operable production stack. The slides are available here.
This continues the story of PagedAttention and vLLM, where KV cache stopped requiring a contiguous region of memory inside a single inference engine. Here, the unit of optimization changes: prefill, decode, and KV cache itself become cluster-wide resources.
- Prefill processes the entire prompt, builds the KV cache, is primarily compute-bound, and determines time to first token
- Decode emits tokens sequentially, depends more heavily on memory bandwidth, and determines inter-token latency
- When both phases share one server, a heavy prefill interferes with steady decode, while both phases are forced into the same hardware and parallelism configuration.
llm-d places them in separate pools
- The scheduler selects a decode worker using prefix-cache state, occupied memory, and queue depth
- If enough work remains uncached, it independently selects a prefill worker
- A sidecar sends that worker a request with
max\_tokens=1; the decode worker then retrieves the computed KV cache through NIXL and continues generation - The interesting shift is that the system is no longer routing only HTTP requests; it is also routing computed model state.
In the project’s own benchmark, the team compared the same 16 H200 GPUs over InfiniBand: four monolithic TP4 replicas versus four TP2 prefill workers and two TP4 decode workers, serving Llama-4 Scout with 5000 input and 250 output tokens. According to the project, P/D delivered noticeably higher per-GPU throughput in the middle of the load curve, especially around 64–128 concurrent requests. At low and extreme load, the curves converged—there is no universal multiplier here.
The story developed in several steps:
- In 2023, PagedAttention made KV cache management more efficient inside one engine;
- In 2024, Splitwise and DistServe showed why prefill and decode could be physically separated, while Mooncake showed how to build a distributed architecture around KV cache;
- llm-d launched on May 20, 2025, and v0.2 added the first reproducible well-lit paths for P/D, cache-aware routing, and wide expert parallelism in July;
- The talk on October 23, 2025 captured an early working system whose central question was no longer “can the phases be separated?” but “how do we select P and D workers and transfer state safely?”;
- In 2026, llm-d joined the CNCF Sandbox, moved beyond mandatory Kubernetes, and explicitly described itself as an inference control plane in v0.8. The P2P KV-cache reuse proposed in the talk became a separate well-lit path on August 15, and on August 17 the project released v0.9.
The speakers supplied the most important caveat themselves: P/D is not for every workload. They recommend beginning with models around 70B or larger, long contexts such as 10k input / 1k output, and sparse MoE architectures. Current vLLM documentation still labels disaggregated prefilling experimental and explicitly warns that disaggregation by itself does not improve throughput. It enables independent tuning of TTFT and inter-token latency; any real gain belongs to the full configuration—topology, router, workload, and a network fast enough to move the cache.
The proper starting point is therefore a workload profile: input/output token distributions, concurrency, TTFT and inter-token-latency SLOs, and the cost of KV transfer. Without those numbers, P/D disaggregation can be either a strong systems optimization or an expensive way to move several gigabytes of cache across the network only to bring it almost back where it started.
#AI #Engineering #Architecture #Software #DistributedSystems #PlatformEngineering