Skip to content
Research Insights Made Simple logo
Episode preparation · paper reviewResearch Insights Made Simple

SGLang: where does the speedup come from?

Up to 6.4× on LM programs and almost no gain on long chat

/ Research Insights Made Simple · SGLang

Slide contents

  1. 1. SGLang: where does the speedup come from?

    Up to 6.4× on LM programs and almost no gain on long chat

  2. 2. Program structure drives the speedup

    Llama-2-7B on A10G · Section 6.2 comparisons

  3. 3. Language and runtime, designed together

    Zheng, Sheng and colleagues: programs know more than servers

  4. 4. Programs over chats, runtime over strings

    Problem → language → runtime → result

  5. 5. Pages store, the tree reuses

    2023 vLLM shared a prefix declared in advance; SGLang finds and keeps any shared prefix itself

  6. 6. From chat to LM program

    Many dependent calls, control flow, structured inputs and outputs

  7. 7. Hard to write, wasteful to run

    Two problems and one answer: a frontend language plus a co-designed runtime

  8. 8. Seven primitives inside Python

    extend, gen, select, fork, join, image, video; Python supplies the rest

  9. 9. The interpreter as an async stream

    Primitives go to a background stream like CUDA kernel launches; reading a result waits

  10. 10. Four patterns of shared prefixes

    Few-shot, self-consistency, multi-turn chat, tree-of-thought

  11. 11. The cache is a tree

    A radix tree holds every request's KV cache; edges are token sequences

  12. 12. Insert, match, split, evict

    How the tree lives under traffic: two chats, few-shot and self-consistency

  13. 13. The cache never fights the batch

    A reference counter protects running requests; zero-count leaves get evicted

  14. 14. Longest shared prefix first

    Sorting the queue by matched prefix equals a depth-first walk of the tree

  15. 15. Program structure helps reuse the cache

  16. 16. Constraints in characters, the model in tokens

    A regex becomes a finite state machine that masks invalid tokens at every step

  17. 17. The compressed automaton jumps forward

  18. 18. Speculation for closed APIs

    One call instead of two: the model runs past the stop, the interpreter matches the remainder

  19. 19. Compare programs, not tokens

    Maximum throughput and single-program latency come from different experiments

  20. 20. The gain depends on the workload

    Llama-2-7B on A10G versus Guidance, vLLM and LMQL; latency down by up to 3.7×

  21. 21. Every piece pays, the tree is free

    Cache hits grow the batch and cut latency; the tree costs 0.3 %

  22. 22. Large models, images, Chatbot Arena

    Tensor parallel keeps the trend; multi-modal up to 6×; in production up to 74 % hits

  23. 23. Where the recipe falls short

    Starvation under greedy scheduling, a single memory tier, distorted probabilities of the compressed automaton

  24. 24. SGLang spread DeepSeek across 96 H100s

    Prefill/decode disaggregation and expert parallelism

  25. 25. HiCache moved caching beyond the GPU

    The paper’s future direction became hierarchical storage

  26. 26. Prefix reuse became a shared concern

    Different data structures serve the same need

  27. 27. What to take from the paper

    Program structure is a resource; the cache is a tree; constraints compress