Skip to content
Research Insights Made Simple
Lab reviewResearch Insights Made Simple

LLM router: a timeout is not a failure

A blind retry after a timeout pays twice, and a model cascade saves only above a computable threshold

/ Research Insights Made Simple · LLM router

Slide contents

  1. 1. LLM router: a timeout is not a failure

    A blind retry after a timeout pays twice, and a model cascade saves only above a computable threshold

  2. 2. A retry after a timeout pays twice

    No answer arrives, yet the provider bills both attempts

  3. 3. Money leaks in three places

    Unknown outcomes, budget reservation, model cascades

  4. 4. Same unknown, same ladder

    “After failure” and “Reasoning effort” seen from the gateway

  5. 5. The gateway budgets before the call

    Policy, cache, tokens, route and reservation come before the provider

  6. 6. A call has three outcomes, not two

    An answer, a confirmed rejection and an unknown lead to different actions

  7. 7. A provider timeout is not a 504

    A domain outcome is HTTP 200 with status; codes are for protocol errors

  8. 8. An unknown outcome keeps the reservation

    Four ledger states; an overdue reconciliation does not release money

  9. 9. The database enforces atomicity, not code

    Checking the limit and writing the reservation are one indivisible step

  10. 10. Two-phase reservations break at settlement

    LiteLLM, Cloudflare, Kong and OpenRouter in their own documentation

  11. 11. A cascade loses money on missed errors

    Four outcomes: the cheap model answered, the verifier accepted or escalated

  12. 12. Precision is a consequence, not a parameter

    One verifier, r = 0.9 and f = 0.1, at three cheap-model levels

  13. 13. The strong model gets the hardest tasks

    θ_c = 0.7, θ_s = 0.9: the chance to rescue an escalated task is 0.90 or 0.667

  14. 14. Cascades pay off only above a threshold

    θ_c = 0.7, θ_s = 0.9, r = 0.9, f = 0.1, ρ = 1: the threshold k at different verifier costs

  15. 15. −46% per request, +1% per solved task

    θ_c = 0.40, θ_s = 0.95, r = 0.20, f = 0.05, k = 2.5, v = 0, ρ = 1

  16. 16. Compare cost per task at equal quality

    Fix the solve rate and ask who reaches it more cheaply

  17. 17. There is no budget ledger yet

    Of 13 requirements: 1 done, 2 partial, 10 not started

  18. 18. Count money by outcome, not by response

    The unknown holds the reserve, the database holds the invariant, the cascade holds a threshold