Prefix-aware routing
Preble: sharing prompt caches across replicas without overheating one
Slide contents
1. Prefix-aware routing
Preble: sharing prompt caches across replicas without overheating one
2. Prefix caches must be visible cluster-wide
Preble supplies the algorithm and measurements; llm-d an engineering path on Kubernetes
3. One radix tree cannot see neighboring replicas
From RadixAttention in one server to a scheduler over many
4. Round-robin recomputes the same repository
Orbit · coding agent · 30–60k-token shared prefix
5. Stickiness and balance pull apart
Spread and recompute; stick and overheat
6. Shared tokens make up 85–97% of prompts
Five public datasets · per-request means · unlimited cache
7. A global tree records where prefixes live
Request-level global scheduler, iteration-level local schedulers
8. Exploit when matched exceeds missed
Otherwise explore: the GPU with the lowest load cost
9. GPU cost is measured as compute time
L: windowed load, M: eviction cost, P: new prefill
10. Hot prefixes get replicated, not overheated
First redirect new requests, then replicate the prefix
11. Cache hits turn requests into decode work
Mix hits and misses on each GPU
12. Batch slots are shared by cache-hit ratio
P priority groups · proportional quota · LRU eviction
13. Testbeds: 2–8 GPUs; the baseline ignores prefixes
SGLang v0.1.12 and vLLM behind a round-robin balancer
14. Gains appear as load grows
Versus round-robin SGLang · mean and p99 latency
15. Azure trace: TTFT drops 2.42×
Tools and video · Azure arrival pattern · four A6000s
16. Each mechanism trims part of the tail
Toolbench · Zipf-1.1 · four A6000s · values read from the figure
17. One scheduler handles 245–2,931 requests per second
245 and 2,931 requests per second · “70–391 GPUs” is the authors' estimate
18. The evidence comes from small testbeds
What the paper shows and what does not follow
19. llm-d learns about caches from event streams
KVEvents → block index → affinity score → load-aware decision
20. Precise indexing gives 0.54 s p90 TTFT
8 pods · 16 H100s · 150 customers with 6k tokens each · 3–60 QPS
21. Stickiness for agents, balance for RAG
The E2 rule treats Orbit's workloads differently
22. A router makes three decisions at once
Where the cache is, what a GPU costs, when to replicate
23. Route by cost, not by match alone
What to measure on Orbit before choosing a router