[2/2] DistServe: Parallelism, Queues, and GPU Balancing (Category #Research)
In the first part of the post, we stopped at separating prefill and decode. That gives us freedom: each stage can receive its own GPUs and distribute the model in its own way. Now that freedom has to become a working configuration.
There are two approaches to parallelism:
- Intra-op divides one operation, such as matrix multiplication, across GPUs. In this context, that means tensor parallelism. An individual step finishes faster, but the GPUs exchange data frequently. The speedup depends on how much communication consumes.
- Inter-op divides model layers into pipeline stages: pipeline parallelism. While one GPU handles the next request, another finishes the previous one. A request still passes through every stage, but the pipeline admits more requests. The cost is idle time when stages are loaded unevenly.
In other words, you can accelerate one request or increase the rate at which the pipeline accepts subsequent ones. Under load, the second option also reduces latency because requests spend less time waiting to enter.
This is where the authors pull queueing theory out of the cupboard. They begin by simplifying prefill: identical prompts, constant processing time D, a Poisson arrival rate R requests/s, and one serving queue without batching. The result is an M/D/1 model:
TTFT = D + R·D² / [2·(1 − R·D)]
This is the mean time: computation plus waiting. The formula works when R·D < 1. If D = 100 ms, at 5 requests/s the mean TTFT is 150 ms, while at 9 requests/s it is 550 ms. The computation itself did not change. The queue grew.
For two GPUs, the authors compare:
intra-op: D/K + R·D² / [2K·(K − R·D)]inter-op: D + R·D² / [4·(2 − R·D)]
K is the intra-op speedup, between 1 and 2 here. The second case assumes two equal stages and negligible communication between them. Each formula applies while its corresponding queue remains stable.
With infrequent requests, intra-op wins because computation is faster. As the arrival rate grows, inter-op can win because of the queue. A very strict TTFT makes individual-request speed critical again. Decode has its own balance among batch size, cache memory, and the TPOT requirement. There is no rule that says “prefill is always one way and decode is always another.”
Real prompts and responses have different lengths, however, and the service needs a specified fraction of requests to satisfy the SLO. One formula for the mean is not enough.
So the team next turns to workload simulation. DistServe uses a compute-time model and a request profile: arrival intensity and distributions of input and output lengths. It enumerates valid parallelism and placement configurations, then uses binary search to find the request rate at which the target SLO attainment still holds. It then selects the number of stage instances for the required load. With a slow network, the prefill and decode options have to be chosen jointly.
The result is a balance for a specific model, hardware, workload, and SLO. When the profile changes, the calculation has to be repeated. The formulas explain the mechanism and the simulator helps choose a configuration; validation on a real cluster remains necessary.
To continue exploring this topic, I plan to read several papers and try llm-d. The rough plan is:
- Splitwise, ISCA 2024 — related work on choosing different hardware for the phases, cost, and energy consumption.
- Sarathi-Serve, OSDI 2024 — another branch: split prefill into chunks and combine them with decode through batch scheduling.
- Mooncake, FAST 2025 — a different scale, where distributed KV-cache storage and reuse become the center of the architecture.
In May 2025, llm-d proposed combining disaggregated serving and cache-aware routing with Kubernetes infrastructure. These works therefore trace a path from separating two stages to managing the computation and state of an entire cluster.
#Research #AI #Architecture #Engineering #DistributedSystems