Cold start
ServerlessLLM: bringing up a new LLM replica faster
Slide contents
1. Cold start
ServerlessLLM: bringing up a new LLM replica faster
2. A new replica helps only once loaded
Three ServerlessLLM mechanisms and one engineering companion
3. Migration moves requests; startup adds capacity
Continuing the Llumnix review and the operations longread
4. The morning peak needs four more replicas
Orbit · 70B chat · 2 replicas at night, 6 by day
5. Loading weights outlasts hundreds of tokens
Why LLM cold starts are slow: download and load
6. Workarounds pay in GPUs, memory or network
Warm pools, DRAM caches and separate storage servers
7. A GPU server already has fast storage
Capacity and bandwidth of local storage tiers
8. The checkpoint is rewritten for reading
Per-GPU partitions, a tensor index and split roles
9. Chunks flow pipelined through every tier
A chunk pool, direct I/O and pinned memory
10. Five techniques push loading to disk bandwidth
Contribution of each optimization · RAID 0 NVMe · OPT models
11. 10 s, not 84: ServerlessLLM loads LLaMA-2-70B
8 × A5000 · RAID 0 NVMe at 12 GB/s · FP16 · cold cache
12. Locality collides with a busy GPU
Four policies for starting model B
13. Move tokens, recompute the KV cache
Multi-round live migration and failure handling
14. The scheduler compares startup-time estimates
Loading q + n/b, migration a × (tin + tout) + b
15. Migration beats preemption on tail latency
4 servers × 4 A40 · 10 Gbps · OPT-6.7B and larger
16. End to end: seconds while GPUs last
Mean model startup latency, s · 300 s timeout
17. Fast startup saves GPUs, doesn’t cure shortages
Load, GPUs per server and the KServe comparison
18. Measured on a small cluster with OPT
What the paper tests and what does not follow
19. Sleep Mode keeps state beyond the weights
Embedded LLM for vLLM · October 26, 2025 · A100
20. By morning, 70B weights must sit nearby
Lower bounds for moving ~140 GB from each tier
21. Pre-warm planned peaks; place surprises by locality
Scaling the 70B chat up each morning and down at night
22. Fast loading does not replace headroom
What the format, migration and scheduler solve — and what they don’t
23. Measure a new replica’s path to tokens
Where are the weights, what does a move cost, what needs warming?