Mapping cluster inference
Nine papers, one platform: from SLOs to operations
Slide contents
1. Mapping cluster inference
Nine papers, one platform: from SLOs to operations
2. A serving cluster is a decision stack
From the product promise to fleet operations
3. Six reviews assemble into one cluster
Continuing the series and two inference longreads
4. Orbit's four workloads carry different promises
16 nodes × 8 H100 · three models · four workloads
5. Count capacity as goodput, not tokens
TTFT, TPOT and the share meeting both
6. Layout depends on tokens per batch
Google · PaLM 540B · 64 TPU v4 · arXiv v1
7. MegaScale-Infer splits attention and experts across nodes
ByteDance Seed · Peking University · arXiv v4 · decode
8. A fleet is replicas, router and queues
A dispatch-time decision is hard to undo
9. Different phases want different pools
Prefill is compute-bound, decode memory-bound
10. Phase splitting does not pay everywhere
Offline work, few GPUs, KV across nodes
11. The KV cache becomes a cluster resource
Moonshot AI · Tsinghua · arXiv v3 · dummy LLaMA2-70B
12. Move KV or recompute: a bandwidth question
Three systems, three answers under different conditions
13. The router weighs cache against load
UC San Diego · arXiv v2 · the E2 rule
14. Rejecting before prefill beats rejecting after it
Load forecasts, priorities and a retry budget
15. Live migration revises the router's decision
Alibaba · OSDI 2024 · downtime around 20–30 ms
16. New replicas first pay for loading weights
Edinburgh · NTU Singapore · OSDI 2024 · arXiv v2
17. Hundreds of adapters share one base model
UC Berkeley · Stanford · SJTU · arXiv v3
18. Tighter coupling widens the blast radius
Losing an Orbit node: 8 GPUs and their state
19. Scale on queues, not GPU utilization
Early signals arrive before complaints
20. Five questions before rebuilding the cluster
Orbit's measurements answer them, not the papers
21. Caching and tuning first, not new topology
Decisions per workload and model, and when to revisit
22. The map combines mechanisms, not results
Different models, hardware, traces and baselines
23. Promise first, topology second
What to measure, split or keep together