vLLM and PagedAttention: Virtual Memory for LLM Serving
How an operating-system idea from the 1960s bought 2-4× serving throughput
Slide contents
1. vLLM and PagedAttention: Virtual Memory for LLM Serving
How an operating-system idea from the 1960s bought 2-4× serving throughput
2. One old idea, one bottleneck
Problem → idea → system → result
3. Generation waits for memory, not math
Prefill is parallel; decoding produces one token per step
4. Where the 40 gigabytes go
65 % model weights, over 30 % KV cache, the rest activations
5. One token costs 800 kilobytes
2 × 5 120 × 40 × 2 bytes; a 2 048-token request is 1.6 GB
6. Three kinds of contiguous-cache waste
Reservation, internal fragmentation, external fragmentation
7. Useful memory: 20 to 38 %
vLLM raises the share of real token states to 96.3 %
8. Pages, bytes, processes: blocks, tokens, requests
The OS vocabulary carried over to the KV cache one to one
9. PagedAttention: attention block by block
The kernel fetches keys and values from non-contiguous blocks by address
10. Block table: logical to physical
The request sees blocks 0-3 in order; in memory they are 7, 1, 3 and nothing
11. New block only when the last fills
Prefill, first step into the free slot, second step into a new physical block
12. One pool instead of private reserves
A free block goes to whoever needs it next
13. Parallel samples: copy on write
One prompt in memory for several answers; only the last block is ever copied
14. Beam search: a block tree, not copies
Candidates share the prefix like forked processes; a system prompt works like a shared library
15. Scheduler: all-or-nothing eviction
FCFS, the newest request is evicted, recovery by swap or recompute
16. One scheduler, many workers
Block tables live in the scheduler; GPU workers receive them with every step
17. 2-4× throughput at the same latency
30 requests per batch instead of 7-14; the gain grows with length and model size
18. Memory sharing: from 6 to 66 %
Parallel sampling, beam search and the shared prefix in numbers
19. The price: a 20-26 % slower kernel
Indirection costs a quarter of one operator; block size 16 is the compromise
20. What to take from the paper
Find the bottleneck, borrow when properties match, pay for indirection knowingly