vLLM and PagedAttention: Virtual Memory for LLM Serving

Alexander Polomodov examines the 2023 vLLM and PagedAttention paper as an example of system-level optimization. Prompt processing uses GPU arithmetic effectively, while sequential generation repeatedly accesses memory. KV cache capacity limits the number of requests that can share a batch. The OPT-13B example shows how little space remains for request state after model weights are loaded, and how reservation and fragmentation further reduce that budget.

The solution borrows paged virtual memory. A table maps each request’s logical blocks to physical GPU blocks, allocated on demand from a shared pool. The PagedAttention kernel reads the cache through that mapping. Diagrams explain sequence growth and why free blocks remain available to other requests. The discussion then turns to shared prefixes: copy on write for multiple answers to one prompt and a tree of blocks for beam search.

When memory still runs out, the scheduler evicts an entire sequence group because its next attention step needs all of its blocks. State can be swapped into CPU memory or discarded and recomputed later. These options spend different resources: data transfer and computation. The vLLM architecture connects a scheduler, a block manager, and GPU workers; tensor-parallel model shards use a consistent mapping of blocks.

The paper’s experiments achieved roughly two to four times Orca’s throughput at comparable latency. Yet the attention kernel itself pays for indirect access. Block size also trades overhead against unused tail capacity. The closing lesson is to identify the actual constraint and evaluate the whole system. Training with more static tensors does not promise the same benefit from dynamic allocation, and the 2023 results cannot replace measurements on a current workload.