vLLM and PagedAttention: Virtual Memory for LLM Serving
How an operating-system idea from the 1960s bought 2–4× serving throughput
Slide contents
1. vLLM and PagedAttention: Virtual Memory for LLM Serving
How an operating-system idea from the 1960s bought 2–4× serving throughput
2. One old idea, one bottleneck
Problem → idea → system → result
3. Generation waits for memory, not math
Prefill is parallel; decoding produces one token per step
4. Where the 40 gigabytes go
65 % model weights, over 30 % KV cache, the rest activations
5. One token costs 800 kilobytes
2 × 5 120 × 40 × 2 bytes; a 2 048-token request is 1.6 GB
6. Three kinds of contiguous-cache waste
Reservation, internal fragmentation, external fragmentation
7. Useful memory: 20 to 38 %
vLLM raises the share of real token states to 96.3 %
8. Pages, bytes, processes: blocks, tokens, requests
The OS vocabulary carried over to the KV cache one to one
9. PagedAttention: attention block by block
The kernel fetches keys and values from non-contiguous blocks by address
10. Block table: logical to physical
The request sees blocks 0-3 in order; in memory they are 7, 1, 3 and nothing
11. New block only when the last fills
① prefill, ② write into the free slot, ③ a new physical block
12. One pool instead of private reserves
A free block goes to whoever needs it next
13. Parallel samples: copy on write
14. Beam search: a block tree, not copies
Candidates share the prefix like forked processes
15. Scheduler: all-or-nothing eviction
FCFS, the newest request is evicted, recovery by swap or recompute
16. One scheduler, many workers
The scheduler keeps the block tables and sends them to workers
17. 2–4× throughput at the same latency
30 requests per batch instead of 7-14; the gain grows with length and model size
18. Memory sharing: from 6 to 66 %
Parallel sampling, beam search and the shared prefix in numbers
19. The price: a 20–26 % slower kernel
Indirection costs a quarter of one operator; block size 16 is the compromise
20. What to take from the paper
Find the bottleneck, borrow when properties match, pay for indirection knowingly