Thousands of adapters
Serving hundreds of LoRA adapters over one base model
Slide contents
1. Thousands of adapters
Serving hundreds of LoRA adapters over one base model
2. One base model for hundreds of adapters
S-LoRA on one machine, Punica across a cluster, Orbit as our example
3. Hundreds of models need not be loaded
Continuing the ServerlessLLM review and episode 32 on vLLM
4. Three hundred merged copies: 4.8 TB
Orbit · 8B base · ~300 team adapters · H100 80 GB
5. An adapter adds xAB to shared xW
W′ = W + AB, rank r ≪ h; S-LoRA targets attention projections
6. Merging wins only with one adapter
S2 · Llama-7B · A10G 24 GB · R = 2, cv = 1 · req/s
7. Batch the base together, adapters apart
All adapters in host memory, only the batch’s adapters on GPU
8. KV cache and adapters share one pool
A page holds H elements: S tokens → S pages, rank R → R pages
9. Adapters reach the GPU during a batch
A forecast from the waiting queue and an asynchronous copy
10. Kernels gather mixed-rank adapters without padding
MBGMM for prefill, MBGMV for decode
11. SGMV: one kernel for a mixed batch
LoRA operator microbenchmark · A100 80 GB · latency
12. Adapter partitioning is aligned with the base
Megatron-LM base partitioning + collectives on small adapter tensors
13. Adapter communication costs less than compute
4 × A100 · 10 adapters · req/s read from the chart
14. Under overload, reject rather than delay everyone
Early abort vs FCFS and LCFS · 6 s first-token SLO
15. 2,000 adapters at nearly the same throughput
One A100 80 GB · synthetic trace · req/s
16. After about a hundred adapters, throughput holds
S4 · Llama-13B · A100 80 GB · req/s read from the chart
17. Punica packs requests onto the fewest GPUs
A new request goes to the busiest GPU that still has room
18. Migration means cancel, then recompute elsewhere
No KV transfer: the new GPU runs prefill again
19. Punica’s throughput holds under any LoRA mix
Llama-2 · A100 · ShareGPT · rank 16 · tokens per second
20. S-LoRA’s numbers come from short synthetic traces
What was measured, and what does not follow
21. Orbit: one 8B pool, not 300 deployments
The host’s proposal; numbers are our estimates and assumptions
22. Shared base settles memory, not the SLO
What was measured, what we synthesized, what remains to test
23. An addition, not a model, separates tenants
What to bring into Orbit, and what to test first?