Request migration
Llumnix: moving running requests with their KV cache across LLM instances
Slide contents
1. Request migration
Llumnix: moving running requests with their KV cache across LLM instances
2. Request placement can change mid-generation
Problem, live migration, one policy, results and Orbit
3. Routing decides once; load keeps changing
Continuing the Preble review and the PagedAttention deck
4. Monday, 9:00: memory grows unpredictably
Orbit · the 70B pool and the coding-agent pool · Monday morning
5. 62% memory load: 8% of requests preempted
LLaMA-7B · vLLM · one A10 · 2,000 requests · 0.42 req/s
6. Free memory exists, but scattered across instances
Four LLaMA-7B instances · 1.9 req/s · each new request goes to the least loaded
7. Moving a running request serves four goals
Load balancing, de-fragmentation, priorities, auto-scaling
8. Finished KV cache is immutable: copy early
Multi-stage copying runs alongside decoding
9. A handshake aborts migration on any surprise
PRE-ALLOC · ACK/ABORT · COMMIT between source and destination
10. 20–30 ms of downtime at any length
Two instances on different machines · LLaMA-7B and LLaMA-30B · up to 8k tokens
11. The global scheduler sees instances, not requests
A llumlet on each instance picks requests and runs the transfer
12. Virtual usage reduces every goal to balancing
Instance freeness F = (M − ΣV) / B
13. One metric drives dispatch, migration and scaling
The freest takes in, the busiest gives away
14. LLaMA-7B served synthetic traces on 16 A10s
4 VMs × 4 A10 24 GB · PCIe 4.0 · 64 Gb/s network
15. Up to 14.8× shorter prefill tail
16 × LLaMA-7B · generated distributions · versus INFaaS++
16. Fragmentation drops from 7.9% to 0.7%
M-M trace · 7.5 req/s · a busy period
17. Urgent requests get headroom; others barely pay
10% high-priority requests · S-S trace · Gamma arrivals
18. Same P99 prefill at 36% lower cost
Up to 16 instances · L-L trace · freeness band [10, 60]
19. Testbed and models bound the paper’s conclusions
A10s and LLaMA-7B, synthetic traces, in-house baselines, failures not covered
20. 40k tokens ≈ 13 GB of KV cache
Assumption: a long 70B-pool request · Llama-2-70B-like KV · FP16 · 400 Gb/s
21. Llumnix covers four of five Orbit scenarios
Morning spike, chat and batch, long prefix, maintenance and node loss
22. Migration revisits placement without replacing neighbouring systems
What Llumnix solves and what it leaves to neighbouring systems
23. Placement is a hypothesis you can revise
What to measure before enabling migration in your cluster?