
LLM router: a timeout is not a failure
A blind retry after a timeout pays twice, and a model cascade saves only above a computable threshold
Slide contents
1. LLM router: a timeout is not a failure
A blind retry after a timeout pays twice, and a model cascade saves only above a computable threshold
2. A retry after a timeout pays twice
No answer arrives, yet the provider bills both attempts
3. Money leaks in three places
Unknown outcomes, budget reservation, model cascades
4. Same unknown, same ladder
“After failure” and “Reasoning effort” seen from the gateway
5. The gateway budgets before the call
Policy, cache, tokens, route and reservation come before the provider
6. A call has three outcomes, not two
An answer, a confirmed rejection and an unknown lead to different actions
7. A provider timeout is not a 504
A domain outcome is HTTP 200 with status; codes are for protocol errors
8. An unknown outcome keeps the reservation
Four ledger states; an overdue reconciliation does not release money
9. The database enforces atomicity, not code
Checking the limit and writing the reservation are one indivisible step
10. Two-phase reservations break at settlement
LiteLLM, Cloudflare, Kong and OpenRouter in their own documentation
11. A cascade loses money on missed errors
Four outcomes: the cheap model answered, the verifier accepted or escalated
12. Precision is a consequence, not a parameter
One verifier, r = 0.9 and f = 0.1, at three cheap-model levels
13. The strong model gets the hardest tasks
θ_c = 0.7, θ_s = 0.9: the chance to rescue an escalated task is 0.90 or 0.667
14. Cascades pay off only above a threshold
θ_c = 0.7, θ_s = 0.9, r = 0.9, f = 0.1, ρ = 1: the threshold k at different verifier costs
15. −46% per request, +1% per solved task
θ_c = 0.40, θ_s = 0.95, r = 0.20, f = 0.05, k = 2.5, v = 0, ρ = 1
16. Compare cost per task at equal quality
Fix the solve rate and ask who reaches it more cheaply
17. There is no budget ledger yet
Of 13 requirements: 1 done, 2 partial, 10 not started
18. Count money by outcome, not by response
The unknown holds the reserve, the database holds the invariant, the cascade holds a threshold