
SGLang: where does the speedup come from?
Up to 6.4× on LM programs and almost no gain on long chat
Slide contents
1. SGLang: where does the speedup come from?
Up to 6.4× on LM programs and almost no gain on long chat
2. Program structure drives the speedup
Llama-2-7B on A10G · Section 6.2 comparisons
3. Language and runtime, designed together
Zheng, Sheng and colleagues: programs know more than servers
4. Programs over chats, runtime over strings
Problem → language → runtime → result
5. Pages store, the tree reuses
2023 vLLM shared a prefix declared in advance; SGLang finds and keeps any shared prefix itself
6. From chat to LM program
Many dependent calls, control flow, structured inputs and outputs
7. Hard to write, wasteful to run
Two problems and one answer: a frontend language plus a co-designed runtime
8. Seven primitives inside Python
extend, gen, select, fork, join, image, video; Python supplies the rest
9. The interpreter as an async stream
Primitives go to a background stream like CUDA kernel launches; reading a result waits
10. Four patterns of shared prefixes
Few-shot, self-consistency, multi-turn chat, tree-of-thought
11. The cache is a tree
A radix tree holds every request's KV cache; edges are token sequences
12. Insert, match, split, evict
How the tree lives under traffic: two chats, few-shot and self-consistency
13. The cache never fights the batch
A reference counter protects running requests; zero-count leaves get evicted
14. Longest shared prefix first
Sorting the queue by matched prefix equals a depth-first walk of the tree
15. Program structure helps reuse the cache
16. Constraints in characters, the model in tokens
A regex becomes a finite state machine that masks invalid tokens at every step
17. The compressed automaton jumps forward
18. Speculation for closed APIs
One call instead of two: the model runs past the stop, the interpreter matches the remainder
19. Compare programs, not tokens
Maximum throughput and single-program latency come from different experiments
20. The gain depends on the workload
Llama-2-7B on A10G versus Guidance, vLLM and LMQL; latency down by up to 3.7×
21. Every piece pays, the tree is free
Cache hits grow the batch and cut latency; the tree costs 0.3 %
22. Large models, images, Chatbot Arena
Tensor parallel keeps the trend; multi-modal up to 6×; in production up to 74 % hits
23. Where the recipe falls short
Starvation under greedy scheduling, a single memory tier, distorted probabilities of the compressed automaton
24. SGLang spread DeepSeek across 96 H100s
Prefill/decode disaggregation and expert parallelism
25. HiCache moved caching beyond the GPU
The paper’s future direction became hierarchical storage
26. Prefix reuse became a shared concern
Different data structures serve the same need
27. What to take from the paper
Program structure is a resource; the cache is a tree; constraints compress