LLM Inference as a Distributed System: From KV Cache and vLLM to Phase Disaggregation

In the textbook definition, inference begins after training: parameters are fixed, an object arrives, and the model returns a prediction. For gradient boosting, an image classifier, or a modest neural network, that definition is close to the operational unit of work. A server can collect independent inputs, run one bounded pass, return fixed-shape results, and release request memory.010101Platforms show this boundary best. I covered the evolution of Uber's Michelangelo on my channel: for eight years the platform lived on predictive models and a feature store, and once LLMs arrived the team had to build a separate layer — a gateway, prompt versioning, quality evaluation, and cost control. Inference turned out to be a different operational object, not one more model type.Knizhny kub · Uber's Michelangelo platform (in Russian)01Platforms show this boundary best. I covered the evolution of Uber's Michelangelo on my channel: for eight years the platform lived on predictive models and a feature store, and once LLMs arrived the team had to build a separate layer — a gateway, prompt versioning, quality evaluation, and cost control. Inference turned out to be a different operational object, not one more model type.Knizhny kub · Uber's Michelangelo platform (in Russian)

An autoregressive LLM is still a function physically, but serving it is no longer one call. The model first processes the entire prompt, emits one token, appends it to context, and invokes almost the whole model again. Output length is unknown. The user expects a stream. Other requests arrive with different lengths and priorities, while short sequences free their slots before long neighbors.

The pivotal difference is the KV cache. Each attention layer retains keys and values for tokens already processed so the next step need not recompute the whole history. Computationally, this is an enormous saving. Systemically, it creates a large state object that grows with context length, concurrent requests, branching samples, and replica count. It must be placed, shared, evicted, transferred, or reconstructed after failure.

Average tokens per second therefore says little about the user's experience. A larger batch can raise aggregate throughput while making interactive requests wait for their first token. A service can start quickly yet emit the rest in bursts. LLM serving needs at least four views of time and capacity.

Goodput puts the SLO back into the capacity definition: only work completed inside the stated TTFT and TPOT limits counts. That is why two configurations with identical aggregate tokens per second can have different operational value.

After tokenization comes prefill. The model receives many prompt positions at once, computes their representations, and populates the KV cache. Large matrix operations can use the accelerator's compute units efficiently. As the input grows, more of the attention and linear-layer work is parallel—until memory behavior and operation shapes impose another limit.

After the first output token has been generated, decode produces the remaining tokens one step at a time. At each step, every active sequence contributes only one new position. Matrix operations narrow, yet the step must read weights and a growing KV cache. At typical low-to-medium load, arithmetic units wait for data from HBM and the phase is bandwidth-bound. A larger batch raises arithmetic intensity but also grows active state and often worsens TPOT.020202I covered the same point from the hardware side on my channel, through Valentin Mamedov's market review. His consumer-versus-datacenter card comparison lands not on peak operations but on memory bandwidth: 1 TB/s against 3 TB/s. For generation, that is the difference that matters.Knizhny kub · where accelerator hardware is heading (in Russian)02I covered the same point from the hardware side on my channel, through Valentin Mamedov's market review. His consumer-versus-datacenter card comparison lands not on peak operations but on memory bandwidth: 1 TB/s against 3 TB/s. For generation, that is the difference that matters.Knizhny kub · where accelerator hardware is heading (in Russian)

Weight memory is only the first budget line

A lower bound for weight memory is straightforward. For a model with P parameters represented with B bits, before runtime buffers:

A 70-billion-parameter model occupies roughly 140 decimal gigabytes in BF16, or roughly 35 GB at four bits before scales, metadata, unquantized layers, and runtime workspace. Freed capacity does not all become new requests: KV state, temporary activations, collective buffers, and headroom to avoid an out-of-memory (OOM) failure remain.

KV memory for one sequence can be approximated as follows, where L is layer count, H_kv is KV-head count, D_h is head dimension, S is bytes per element, and T is processed tokens:

The factor of two is K plus V. Under Multi-Query Attention (MQA), all query heads share one K/V set; under Grouped-Query Attention (GQA), each of several groups has one. Model architecture therefore fixes the cost of each active token in advance. Runtime selection cannot rely on parameter count alone: two similarly sized models with different KV-head counts behave differently at long context.

Four distributions matter more than one average length

• —Input length controls work before the first token and initial KV state.
• —Output length controls serial passes and the lifetime of that state.
• —Concurrency controls how much weight movement can be amortized by a batch.
• —Prefix overlap controls how much prompt work can be reused.

These axes interact nonlinearly. Ten short requests may fill tensor cores better than one long request, while ten long outputs retain KV orders of magnitude longer. High prefix overlap removes compute only after warmup and only when routing finds the right replica. A concurrency burst changes more than the queue: it simultaneously increases batch size, arithmetic intensity, and memory pressure. The profile therefore cannot be reduced to three separate means: input sequence length (ISL), output sequence length (OSL), and queries per second (QPS). It needs their joint trace with temporal correlations intact.

Long-document RAG, short-turn chat, code generation, and reasoning with thousands of output tokens are not one workload. An average of 2K input and 500 output tokens can hide rare 100K-context requests that hold memory and break the TTFT tail. A credible benchmark must replay the joint length and arrival distribution instead of looping over one string.

Most techniques now exposed as engine flags did not begin with vLLM. KV caching had long supported autoregressive decoding. In 2019 Noam Shazeer proposed multi-query attention, letting query heads share K/V to reduce bandwidth. In 2023 GQA occupied a middle point: several KV groups rather than one or one per query head.

Weight sharding, pipeline execution, specialized GEMMs, fusion, and low-precision formats evolved alongside them. GPTQ compressed weights with approximate second-order information, SmoothQuant moved outlier difficulty from activations into weights for W8A8, and AWQ protected a small share of salient channels. These methods solve different problems. Weight-only INT4 reduces weight traffic and capacity but does not promise a fourfold speedup: without a fast hardware kernel, unpacking and conversion consume the saving.030303I like that this effect reproduces even on a laptop. I covered a long practical piece on local serving on my channel: it shows plainly that a smaller quant is not necessarily faster — i-quants demand more compute, and the bytes you saved go back into arithmetic.Knizhny kub · getting more out of local LLMs (in Russian)03I like that this effect reproduces even on a laptop. I covered a long practical piece on local serving on my channel: it shows plainly that a smaller quant is not necessarily faster — i-quants demand more compute, and the bytes you saved go back into arithmetic.Knizhny kub · getting more out of local LLMs (in Russian)

FlashAttention opened another line in 2022: make exact attention faster without changing its mathematical output by tiling and reducing HBM ↔ SRAM traffic. The lesson was broader than attention. Nominal FLOPS do not determine wall time when the GPU spends most of the cycle moving data.

Orca changed the scheduling unit

Traditional serving often formed a static batch and waited for every request to finish. A 20-token completion next to a 500-token completion therefore waited through 480 irrelevant iterations while new work could not enter. At OSDI 2022, Orca introduced iteration-level scheduling and selective batching.

Modern continuous batching inherits that idea: completed sequences leave, new ones enter, and paused work may be preempted. This attribution matters. vLLM implemented and popularized continuous batching in an accessible engine, but it did not invent the iteration-level scheduling principle.

In spring 2023 LMSYS served Vicuna and Chatbot Arena on a constrained university GPU fleet. According to the original vLLM announcement, the Hugging Face Transformers backend became a bottleneck as traffic grew. The UC Berkeley team built vLLM and integrated it with FastChat. The practical question was not how to speed up one GEMM, but how to keep more live requests on the same GPUs.

KV-cache management was the obstacle. Future output length is unknown, yet a traditional allocator either reserves a contiguous maximum or accumulates external fragmentation after differently sized requests depart. The launch post measured 60–80% memory waste from reservation and fragmentation in prior systems. That figure belongs to the authors' workloads and implementations, but the mechanism is general.

PagedAttention borrows the virtual-memory analogy. A sequence sees contiguous logical KV blocks, while a block table maps them to arbitrary physical GPU blocks. Capacity is allocated as generation proceeds, so free blocks remain in a common pool and can immediately hold another request's KV state; no request reserves space for an unknown maximum in advance. Waste largely remains in the final partially filled block. Parallel samples can reference the same prompt pages and diverge through copy-on-write.

Memory efficiency does not make one attention operation magically faster. It admits a larger effective batch, amortizes weight reads, and raises GPU utilization. The SOSP paper reported 2–4x throughput at comparable latency versus FasterTransformer and Orca, with larger gains for longer sequences, larger models, and multi-output decoding such as parallel sampling and beam search, where branches can share blocks for their common prefix. The launch post reported up to 24x over a basic Hugging Face backend and up to 3.5x over TGI. Those are different baselines; they cannot be collapsed into “vLLM is 24x faster.”

vLLM's second contribution was organizational. An open API, broad model support, and one serving engine gave researchers a place to integrate new kernels, schedulers, and KV-transfer paths. It became to inference what a widely used runtime becomes to a language: not the only possible implementation, but common ground on which ideas reach operations faster.040404One layer down, this plot has already played out. I watched the PyTorch documentary and wrote about it on my channel: a tool for fast experiments became the standard through developer experience, ecosystem, and neutral governance rather than benchmarks. vLLM is repeating that trajectory at the serving layer.Knizhny kub · the PyTorch documentary (in Russian)04One layer down, this plot has already played out. I watched the PyTorch documentary and wrote about it on my channel: a tool for fast experiments became the standard through developer experience, ecosystem, and neutral governance rather than benchmarks. vLLM is repeating that trajectory at the serving layer.Knizhny kub · the PyTorch documentary (in Russian)

A page of engine flags suggests independent speedups. The real system is a stack of interacting transformations. Quantization shrinks weights and can expose the KV cache as the next bottleneck. Prefix caching saves prefill but creates key skew and a need for locality-aware routing. A large batch raises tensor-core utilization but retains more KV state and increases the time between tokens for the slowest requests, such as the P95/P99 tail of TPOT.

Roofline ties a speedup to the bytes it actually removes

A useful test for any technique begins with arithmetic intensity: operations executed per byte read from memory. The performance ceiling R is the smaller of the accelerator's peak compute and intensity multiplied by memory bandwidth:

The difference is how much useful work follows one weight read. During a long prefill, a weight tile is loaded and immediately applied to many prompt positions: one HBM transfer serves many operations, so the point sits toward the right of the chart. During next-token generation, each sequence contributes only one new position; a similar volume of weights serves far fewer operations, so memory bandwidth more often sets the speed. Batching several sequences reuses the loaded weights and moves decode to the right, but consumes more KV capacity and makes requests wait for one another. Aggregate tokens per second can therefore rise while TPOT gets worse.

Quantization pays only when fewer bytes match the workload's limit and the GPU can execute the chosen format efficiently. Weight-only W4A16 often helps small-batch decode because it reduces weight traffic. W8A8 or FP8 may accelerate prefill matrix operations when suitable tensor cores exist and outliers are calibrated. KV quantization shrinks growing request state but touches attention on every step and needs a separate long-context quality evaluation. None of these effects follows from bit width alone.

Kernels, fusion, and graphs remove different overheads

A specialized GPU kernel is a program the GPU launches for one operation; it defines data layout, HBM-to-SRAM traffic, vectorization, and numeric format. Operation fusion combines several consecutive operations into one such kernel. The intermediate tensor no longer needs an HBM write and read, and the separate launch and synchronization disappear. CUDA Graphs instead reduce the launch time of a repeated kernel sequence.

Deep fusion means combining a long operation chain into one kernel. The longer the chain, the more registers and shared memory it needs, so fewer blocks may execute concurrently. Such a kernel is also harder to adapt to new model architectures. Engines therefore retain multiple execution paths and dispatch by shape. A variable length, rare branch, new adapter, or uncommon batch size may return execution to an eager path.

The parallelism strategy determines what accelerators exchange and how often

There are two fundamentally different ways to distribute work. TP, PP, and EP split one model pass across accelerators: a request completes only through their joint execution. Replicas instead perform complete passes and split independent requests. The choice therefore changes the network path, synchronization points, and source of idle time.

With tensor parallelism, parts of one matrix operation run at the same time. In the variant shown, each GPU computes a partial result and all-reduce sums the partials before the next layer. This lowers weights and compute per GPU, but makes a collective part of nearly every layer. During a short decode step, interconnect latency can consume the compute saved by sharding.

Pipeline parallelism sends a request through consecutive layer groups on different GPUs. The batch is split into microbatches so stages can run concurrently. The edge stages idle while the pipeline fills and drains, and unbalanced layer groups create additional bubbles. PP therefore works best with enough microbatches and a predictable workload.

Expert parallelism applies to MoE models. A router selects experts for each token; token data travels to the GPUs that own those experts and returns after computation. The network is therefore used twice. If one expert is selected more often, it queues work while other GPUs remain underused: activating fewer parameters does not by itself balance the system.

A replica instead receives an independent request and performs the complete pass within its accelerator group. Replicas do not exchange data on every token; a load balancer only chooses a destination for a new request. This scales request throughput and isolates tenants or failures, but every replica group holds another copy of the model. TP, PP, or EP may still be used inside a replica.

The optimal layout may differ between prefill and decode. A long prompt can keep a wide TP group busy, while a short decode step may spend more time in collectives than it saves in compute. This is a reason to evaluate phase disaggregation, not proof that it will help. First benchmark layouts in which both phases remain in one pool: TP alone, TP with PP, and—for MoE—EP as well. Separate pools are justified only when two specialized layouts outweigh KV transfer and the additional queue.

Multi-LoRA turns one base model into a multi-tenant service

Multi-LoRA serves several fine-tuned variants of one base model at the same time. The server keeps one copy of the base weights and applies the selected LoRA adapter—a small set of low-rank deltas—for each request. That saves HBM relative to one replica per tenant and lets different adapters share a batch. The cost appears in scattered reads from different deltas, loading cold adapters, more complex kernels, and fairness scheduling. If the active adapter set is large or traffic is highly skewed, the adapter cache and queue become the next bottleneck. The mechanism preserves the behavior of each adapter, but it cannot fix a tokenizer mismatch, a different base-model revision, or incompatible quantization formats.

Offload and tiered caches trade capacity for latency

Once HBM is full, weights, adapters, or KV can be moved to host memory, local storage, or a remote cache. The service can then retain more state than GPU memory can hold, but off-device data is slower to access. It must be brought back into HBM before computation, so lookup, queueing, and the RAM, storage, or network-to-GPU transfer path determine the cost. For a large prefix, one fast transfer may beat recomputation; for a short continuation, finding a remote block can take longer than rebuilding it.

In a cluster, locality conflicts with load balance. A router that always selects the hot cache overloads a popular replica; one that always selects the shortest queue destroys computation reuse. Distributed inference systems with KV-cache-aware routing address this conflict. Preble is a research serving system for scheduling requests with shared prefixes; llm-d is an open Kubernetes-native stack whose router considers both prefix-cache state and replica load. Their scope differs, but the principle is the same: choose a destination from cache locality and queue pressure together rather than applying a blanket “cache first” rule. The right result is the goodput improvement after misses, evictions, block duplication, and index recovery time are included.

Chunked prefill attacks stalls, not the amount of compute

When a scheduler inserts a long prompt between steps of active completions, TPOT spikes. SARATHI split a large prefill into chunks and inserted one into a decode-maximal batch. At OSDI 2024, Sarathi-Serve reported serving-capacity gains under tail-latency SLOs: 2.6x for Mistral-7B on one A100, 3.7x for Yi-34B on two A100s, and up to 5.6x for Falcon-180B with pipeline parallelism against the authors' selected baselines. These are three experiments, not multipliers for one deployment.

Prefix reuse turns computation into placement

SGLang introduced RadixAttention: retain KV for shared prefixes in a radix tree and reuse them across calls in a language-model program. A few-shot template, common system instruction, or shared RAG document can avoid most prompt computation. On one GPU the policy is lookup plus eviction. In a cluster the request must reach a replica that already owns the prefix without turning that replica into a hotspot. Preble adds global scheduling based on recomputation cost and fairness.

Speculation changes the number of serial target passes

Speculative decoding uses a cheap draft model to propose several tokens; the target verifies them in parallel and accepts a valid prefix without changing its target distribution. Medusa adds several heads to the target; EAGLE predicts penultimate-layer features. Their economics depend on acceptance rate, draft cost, verification batch, and the original bottleneck. If the target is already compute-bound on a large batch, extra verification can reduce throughput even while improving one request's latency.050505I covered a sobering measurement of that economics on my channel. A Google study describes shipping completions in its internal IDE: speculative decoding sits next to an adaptive cache, the cache serves about 35% of requests, and median latency drops by only 9%. A technique is judged at the end of the funnel, not by its own metric.Knizhny kub · AI features inside Google's internal IDE (in Russian)05I covered a sobering measurement of that economics on my channel. A Google study describes shipping completions in its internal IDE: speculative decoding sits next to an adaptive cache, the cache serves about 35% of requests, and median latency drops by only 9%. A technique is judged at the end of the funnel, not by its own metric.Knizhny kub · AI features inside Google's internal IDE (in Russian)

• •Freeze a baseline with identical model, sampling policy, and request trace.
• •Change one layer and measure TTFT, TPOT, goodput, memory, and quality together.
• •Test interactions: quantization × kernels, chunk size × scheduler, cache × router.
• •Never multiply vLLM, Sarathi, EAGLE, and cache figures from different papers.

By late 2023 a shared GPU pool was coupling two decisions that operators wanted to make independently. Fast TTFT for a long prompt benefits from high compute and one tensor/pipeline-parallel plan. Stable TPOT needs memory bandwidth, another batch shape, and sometimes a cheaper accelerator. One compromise plan either harms one phase or reserves excess capacity for both.

Disaggregated inference gives prefill and decode independent worker pools. A prefill worker constructs KV, state moves to a decode worker, and that worker continues the sequence and streams output. This is not pipeline parallelism. PP places layers of one model pass on different devices; phase disaggregation assigns whole temporal phases of a request to separate replicas of the model.

Papers often abbreviate this architecture as P/D: P (prefill) processes the input context and builds the KV cache, while D (decode) generates tokens step by step from that state. The full phase names are used below whenever the abbreviation would hide the mechanism.

Splitwise made the first explicit phase-splitting proposal

Microsoft's Splitwise preprint appeared on 30 November 2023. It characterized compute-intensive prompt computation and memory-intensive token generation, proposed placing them on different machines, selecting GPU generations independently, and transferring state over a fast cluster interconnect. Its objectives included throughput, cost, and power.

On 18 January 2024 DistServe independently framed the problem through SLOs: remove interference, select phase-specific resources and parallelism, and maximize goodput under TTFT and TPOT constraints. Its experiments reported up to 7.4x more served requests or a 12.6x tighter SLO than selected systems while meeting constraints for more than 90% of requests. Those results describe large configurations, not a promise for two GPUs.

The full request path is longer than the KV handoff between pools

The ingress router first chooses the model, adapter, and eligible pool, then checks whether a suitable prefix already exists. A miss enters the prompt-processing queue. After tokenization and prompt processing, the worker records the block map, numeric format, positional parameters, and continuation point. Only then can it select a token-generation replica based on free slots, topology, and existing state. The receiver confirms KV ownership before streaming begins; otherwise a failure between transfer and first token leaves the system without an unambiguous owner for the request.060606Classic services reached the same conclusion earlier. When I went through Meta's infrastructure on my channel, the most useful part was exactly the end-to-end request path: a point of presence, long-lived connections into a private WAN, a load balancer, a frontend function, and dozens of backends. LLMs differ in one respect that changes everything: state travels that path too.Knizhny kub · the end-to-end request path at Meta (in Russian)06Classic services reached the same conclusion earlier. When I went through Meta's infrastructure on my channel, the most useful part was exactly the end-to-end request path: a point of presence, long-lived connections into a private WAN, a load balancer, a frontend function, and dozens of backends. LLMs differ in one respect that changes everything: state travels that path too.Knizhny kub · the end-to-end request path at Meta (in Russian)

A lower bound on transfer time is KV size divided by effective network bandwidth, plus queueing, connection setup, and protocol metadata. Effective bandwidth is almost always below the link's label: requests share the channel, blocks arrive unevenly, and the protocol may introduce extra copies. Block-wise or layer-wise transfer lets D start before P fully completes, but introduces backpressure: generation must not run ahead of the layer state it needs.

Rate matching begins with two service rates: input tokens per second that P turns into KV, and active sequences that D can advance without violating TPOT. A sufficient mean rate does not guarantee stability. A burst of long prompts fills the transfer buffer, while long outputs retain D slots after P has finished its work. The system needs admission control, quantile headroom, and per-queue scaling, but those decisions must coordinate so that a new P worker does not overload the old D pool.

Recomputation remains a first-class recovery policy. For a short prompt, rebuilding KV on D may be cheaper than locating a remote copy; for a long shared prefix, replication or a durable cache tier is better. The policy must include block age, probability of a later turn, network cost, and accelerator queueing. Phase separation thereby turns a local memory object into a distributed contract: who owns a block, who may delete it, and which event means the request can safely continue.

Mooncake made the KV cache the center of architecture

In June 2024 Moonshot AI published Mooncake, the platform behind Kimi, combining disaggregated prefill with a distributed KV cache and tiered memory. The story was no longer merely about two GPU pools. KV became a data-plane object: blocks need indexing and placement across HBM, host memory, and storage, while the router needs cache affinity as well as load.

P/D-Serve described operating across tens of thousands of xPUs: dynamic prefiller/decoder ratios, rejection forwarding, and optimized device-to-device transfer. By 2025–2026 Dynamo and llm-d were packaging the same ideas into open orchestration layers above vLLM, SGLang, and TensorRT-LLM. The inference engine executes the model; a distributed runtime owns routing, cache state, scaling, and recovery.

Disaggregation removes local interference and creates a mandatory handoff. State volume grows linearly with input tokens and the model's KV coefficient. If a P node produces tens of gigabytes of KV faster than the fabric can deliver it to D, the queue merely moves from the model scheduler to transport. TTFT now includes both pools, metadata coordination, and KV movement.

DistServe therefore places phases with bandwidth in mind, Mooncake builds a separate transfer engine, vLLM exposes connectors over NIXL, Mooncake, and LMCache, and Dynamo separates request, control, and state planes. With NVLink/NVSwitch inside a domain and RDMA across nodes, a handoff can overlap with computation and layer-wise transfer. On ordinary Ethernet or a congested fabric the advantage can disappear.

Two production lines must be rate-matched

A P:D ratio cannot be copied from documentation. It depends on input/output ratio, reasoning length, cache hits, and target SLOs. If prefill produces KV faster than decode frees slots, D becomes the queue and P idles. For short generations, the prefiller can be hot. Scaling one phase changes pressure on the other, so an autoscaler needs arrivals, tokens in flight, and transfer backlog—not utilization alone.

Multi-turn agents break a one-way pipeline

On turn one, P → D is natural. After output, the KV state lives at D. On the next turn a user appends a small fragment: a canonical P/D design may send old state back to P for append-prefill and then forward it to D again. In an agent loop with tool calls, those bounces can dominate compute. The 2026 PPD preprint sometimes runs append-prefill on D; Load-Aware Prefill Deflection moves some new prefills to D when queueing and transfer cost more than local interference. These are hybrids, not repudiations of phase specialization.070707The market shows this is no longer theory. I covered SemiAnalysis on accelerator rentals on my channel: prices are driven not by training but by production agent inference, where one task expands into a tree of calls. Such workloads hit peak concurrency harder than they hit total token volume.Knizhny kub · the GPU rental market and agentic load (in Russian)07The market shows this is no longer theory. I covered SemiAnalysis on accelerator rentals on my channel: prices are driven not by training but by production agent inference, where one task expands into a tree of calls. Such workloads hit peak concurrency harder than they hit total token volume.Knizhny kub · the GPU rental market and agentic load (in Russian)

Failure is a state-ownership question

If P dies before a confirmed handoff, the prompt can be replayed. If D dies after transfer, the system must locate a KV copy, restore it from another tier, or recompute the prompt. A stale cache index sends an affinity-routed request to an empty replica. Scaling down a node that holds a hot prefix can spike TTFT without any traffic growth. A production design must define block ownership, metadata lifetime, handoff idempotency, and partial-transfer behavior explicitly.

The paper Revisiting Disaggregated LLM Serving adds another caveat: performance and energy gains are not guaranteed and depend on workload, transfer path, and baseline. In its measurements, phase-specific frequency control did not offset the system's extra energy. “Disaggregation saves GPUs” is therefore a hypothesis until a trace, topology, and SLO prove it.

A modern engine such as vLLM or SGLang combines continuous scheduling, optimized attention/GEMM/MoE kernels, multiple weight and KV formats, prefix caching, speculative decoding, and several parallelism strategies. Above it, a cluster runtime may add cache-aware routing, tiered memory, autoscaling, and P/D. Models are increasingly designed for serving too: GQA/MLA reduce KV, MoE reduces active parameters, and multi-token prediction supports speculation.080808At the other end of the scale it looks the same. I covered Maxime Labonne's talk from Liquid AI on my channel: counting nominal operations is not enough, and an architecture has to be validated by profiling prefill, decode, and memory on the target device itself. Co-design is not only a cluster story.Knizhny kub · Liquid AI on training small models (in Russian)08At the other end of the scale it looks the same. I covered Maxime Labonne's talk from Liquid AI on my channel: counting nominal operations is not enough, and an architecture has to be validated by profiling prefill, decode, and memory on the target device itself. Co-design is not only a cluster story.Knizhny kub · Liquid AI on training small models (in Russian)

This does not mean every deployment should reproduce a hyperscaler's architecture. A single node with a colocated engine avoids KV transfer and has fewer failure modes. Chunked prefill often provides enough TPOT control. Prefix caching may matter more than P/D when the workload has a huge shared prompt. Disaggregation pays when scale can independently fill both pools and TTFT and TPOT genuinely need different plans.

Encoder disaggregation extends the idea but changes the transferred object

A multimodal LLM adds a vision or audio encoder that creates embeddings before the text decoder performs prefill and generation. vLLM's Disaggregated Encoder documentation describes an E/P/D topology: the encoder gets its own pool and transfers an embedding cache into prefill and decode instances. This reinforces phase specialization but does not make P/D details universal for diffusion or speech. Each family has its own state, recomputation boundary, and step granularity.

The next frontier splits attention, FFN, and experts

2026 preprints model deeper attention/FFN disaggregation and heterogeneous accelerators for the same reason: phase arithmetic intensity and communication differ—see Song et al. and the limits of AFD. The finer the boundary, however, the more often tensors cross the network and the more the system depends on a specialized interconnect. This is a co-design research direction, not a recommendation to build four-way disaggregation today.

Model → local engine → KV plane → router → capacity planner → SLO

The central shift is that inference is no longer a library function at the end of an ML pipeline. It is a distributed system in its own right: the model fixes state size and viable formats, hardware fixes the roofline and topology, the runtime fixes work order, and the product fixes request distributions and SLOs. Optimizing one layer works only until the next bottleneck.

The practical sequence begins with production replay. Capture the joint input/output length distributions, arrival process, concurrency, shared-prefix rate, multi-turn distance, and agent pauses. Then define SLOs and the cost of violation. Only then compare variants while holding model, sampling, quality, and request set constant.090909The same principle governs model choice, not only engine choice. I covered an Artificial Analysis comparison shown in a 2025 talk: there is no single frontier — intelligence, open weights, cost, and speed are separate. In those measurements, o4-mini (high) took more than forty seconds to answer, while GPT-4.1 took about 4.7 seconds; across a thirty-step agent loop, that difference determines the end-to-end delay.Knizhny kub · Artificial Analysis on the AI frontiers (in Russian)09The same principle governs model choice, not only engine choice. I covered an Artificial Analysis comparison shown in a 2025 talk: there is no single frontier — intelligence, open weights, cost, and speed are separate. In those measurements, o4-mini (high) took more than forty seconds to answer, while GPT-4.1 took about 4.7 seconds; across a thirty-step agent loop, that difference determines the end-to-end delay.Knizhny kub · Artificial Analysis on the AI frontiers (in Russian)

A minimum comparison protocol

• •Establish a baseline on one colocated runtime without experimental flags.
• •Split TTFT into queueing, prompt processing, transfer, and first-byte delivery.
• •Split TPOT by batch size, active KV length, preemption, and new-prefill interference.
• •Replay a representative trace to stable P50/P95/P99, including warm-up and cold starts.
• •Validate quality after quantization/speculation and resilience after worker loss.
• •Compute goodput per GPU and fully loaded cost, not the best isolated tokens/s.

The scorecard should not crown a universal winner among vLLM, SGLang, and TensorRT-LLM. It should prove that a particular runtime, format, scheduler, and topology sustain your profile. On one platform vLLM may have the required model coverage, SGLang the best prefix reuse, and TensorRT-LLM the strongest specialized kernel; after a model update the order can change. The durable contract is a reproducible trace and SLO, not the project name.

For a small fleet, the sensible default is intentionally boring: one validated model, a colocated engine, continuous batches, an appropriate numeric format, chunked prefill for long inputs, and a prefix cache where repetition is measured. Cache-aware routing comes with multiple replicas. P/D follows only when the profile shows a stable TTFT/TPOT conflict and the fabric demonstrably transports KV more cheaply than recomputation.