Experts across a cluster
MoE models across a cluster: attention/expert disaggregation, micro-batch pipelines and M-to-N exchange
Slide contents
1. Experts across a cluster
MoE models across a cluster: attention/expert disaggregation, micro-batch pipelines and M-to-N exchange
2. MoE across a cluster moves the bottleneck
MegaScale-Infer (ByteDance Seed) and the DeepSeek-V3/R1 system overview
3. Experts add a new partitioning axis
From tensor and pipeline partitioning to expert parallelism
4. The coding assistant’s MoE needs several nodes
16 nodes × 8 H100 80 GB · NVLink inside a node · InfiniBand 400 Gb/s
5. Sparsity splits the batch across experts
The paper’s example: Mixtral 8x22B on A100, bf16
6. Expert parallelism needs two all-to-all exchanges
Tokens travel to their chosen experts and back
7. Attention and experts move to separate nodes
Disaggregated expert parallelism: M attention replicas, N expert nodes
8. Micro-batches fill idle time, ping-pong style
m micro-batches alternate between attention and experts
9. Deployment plans are searched under an SLO
Attention and expert TP, replica count, micro-batches, batch size
10. Attention wants memory; experts want compute
Per-price specifications · prices normalized to L20
11. M-to-N traffic needs its own library
No GPU-to-CPU copies, group setup or GPU synchronization
12. At 256 KB, p99 latency drops 92.9%
Microbenchmark: 8 senders, 8 receivers, one 200 Gb/s NIC per GPU
13. Decode gains reach 1.90× over TensorRT-LLM
8 nodes × 8 Ampere 80 GB GPUs · TBT ≤ 150 ms · bf16
14. Per-token latency does not shrink
Every layer adds a cross-node exchange
15. H20/L40S pairs beat homogeneous clusters on cost
Attention on H20, experts on L40S · per unit price and power
16. Disaggregation, not just networking, drives the gain
Ampere ablations: disaggregation, M2N, micro-batches, attention replicas
17. In production, expert load is uneven
Nearly 10,000 GPUs · 1.5–2.0× lower cost, as reported by the authors
18. DeepSeek stretches expert parallelism over 18 nodes
8 of 256 experts per layer · prefill EP32 · decode EP144
19. DeepSeek’s one-day load peaked at 278 nodes
Feb 27, 2025 12:00 → Feb 28, 2025 12:00, UTC+8 · 8×H800 nodes
20. Gains depend on decode and testbed conditions
What was measured and what does not follow
21. For Orbit, first count tokens per expert
Assumptions: DeepSeek-V3 shape, 8 of 256 experts; H100 ≈ Table 3’s H800
22. Disaggregation pays off only when balanced
Per-expert batch, Ta ≈ Te, communication shorter than compute
23. Experts force cluster planning, not node planning
What Orbit adopts and what it tests first