Operating LLM Inference: What to Promise and How to Serve Requests Under Overload

The same inference service can be described two honest ways. The engineer shows a curve: so many tokens per second at saturation, ahead of the competition, twice as fast as last quarter. The product shows something else: the share of conversations where the answer took more than three seconds to start has doubled. Both measurements are correct, and they barely relate to each other.

The difference is not measurement rigour but what is being measured. Throughput describes how much work the machine completes when it is fully loaded. A product does not live in the average; it lives in the tail — in what happened to the requests unlucky enough to queue behind somebody's very long context. A benchmark without a threshold describes hardware. A promise describes behaviour at the boundary.

You do not have to invent the format of that promise. The industry has already written it down. MLPerf Inference v5.1 rates a system on Llama 3.1-8B not by raw speed but by the load it sustains inside two bounds. In the Server scenario that means TTFT at or below 2 seconds and TPOT at or below 100 milliseconds. In the newer Interactive scenario the bounds tighten to 0.5 seconds and 30 milliseconds.

MLCommons translates those thresholds into a human unit itself: 100 milliseconds per token is roughly 480 words per minute, noticeably faster than people read, while 30 milliseconds is about 1,600 words per minute. The second scenario exists for cases where responsiveness is the point: coding assistants and real-time creative tools.

By October 2026 the format has settled in. In the MLPerf Inference v6.1 round of 16 September 2026 every language model carries its own pair of thresholds: Llama 2 70B in the interactive scenario gets 450 milliseconds to the first token and 40 milliseconds per token, DeepSeek-R1 gets 1.5 seconds and 15 milliseconds. A reasoning model may think longer before it starts, but must print faster: thresholds follow the scenario, not the model in general. In the same announcement MLCommons said that for the datacenter MLPerf Inference will give way to MLPerf Endpoints — a benchmark that loads the system through an API, the way a client sees it.

Researchers gave this measure its own name: goodput. In DistServe (OSDI 2024) it is the maximum request rate at which a target share of requests — say, 90 % — meets the SLO, per accelerator provisioned. In the paper's opening example a system that runs both phases on one accelerator keeps about 1.6 requests per second inside the thresholds, while a system with disaggregated phases keeps 3.3 requests per accelerator. Same model and hardware; what differs is the share of work that kept the promise.

Why the pace of generation has a threshold at all comes down to a simple observation: text faster than reading does not make an answer better. The authors of the Andes preprint take reading speed as 4.8 tokens per second and listening speed as 3.3, and score an answer by how far delivery lags behind that pace. It is a preprint, not a standard, but the logic is useful: outrunning the reader buys nothing, lagging behind is noticeable. Past some point faster generation gives the product nothing, while time to first token always matters.

promise = a threshold on the start of the answer + a threshold on the pace of generation + the share of requests that stay inside both

Everything that follows is about where the breach of that promise comes from: why one latency curve is not enough, where the tail actually lives, what capacity is measured in, how the whole thing breaks, and what to do instead of failing. How the server itself works — how it forms batches, where the KV cache lives and why phases get split — is covered in the article on inference as a distributed system; here it matters only where it affects the promise.

For a generative service, "slow" splits into at least three different events. The user presses a button and nothing happens — that is time to first token. The answer starts but prints slower than it reads — that is the pace of generation, the time per output token. The whole answer does not arrive before the caller times out — that is end-to-end latency.

Averaging these into a single curve loses information rather than simplifying it. Time to first token grows with the queue and with input length; the pace of generation drops when the machine gains neighbours; end-to-end latency can grow simply because answers got longer. Three causes, three knobs.

vLLM already separates delays and queue state. The names below connect each metric to a question about the service.

A dedicated queue-time metric is not a detail. It separates what the model is responsible for from what the scheduler and the capacity are responsible for. If your dashboard shows one latency curve, that is a team decision, not a tooling limit — the split has already been made for you.

Engineering intuition says latency grows when capacity runs short. That is half true, and the other half explains most of the unpleasant incidents.

The classic approximation for mean waiting time is Kingman's formula, published in 1961. It factors waiting into three multiplicands: utilization, variability and service time.

wait ≈ [ρ / (1 − ρ)] × [(c²arrival + c²service) / 2] × mean service time

The managerial reading is that waiting grows for three independent reasons: arrivals become burstier, service times spread out, utilization approaches one. Buying capacity only addresses the third.

In inference the first two are extreme, and that is what separates it from an ordinary service. The same endpoint serves a request with a few hundred context tokens and one with hundreds of thousands; answers run from one line to ten screens. The spread of service time is measured in orders of magnitude, not percentages. That is why a cluster "only 60 percent loaded" confidently produces an unacceptable tail — and why an extra accelerator barely moves it.

Real traces show it too. The authors of Splitwise (ISCA 2024) published traces of two Azure services: the coding assistant has a median input of 1,500 tokens and a median output of 13; the conversation service has 1,020 and 129, with an almost bimodal output length. The BurstGPT (KDD 2025) dataset — 213 days of traces from regional Azure OpenAI services — shows the second factor from another angle: burstiness changes sharply during working hours, and the failure rate of the ChatGPT conversation service in these traces stays above 5 %. The authors tie such failure spikes to memory bottlenecks, which the next section returns to.

Boundary of applicability. Kingman's formula approximates a single-server queue and is most accurate near saturation. An inference server is not a single-server queue: it batches requests, preempts them and reschedules on every iteration. Here the formula is a thinking tool, not a capacity calculator.

A worked example: the tail-latency budget

The promise from the first section can be split into a budget, and the budget turned into an alert threshold. Suppose the product promises that 95 % of interactive requests see the first token within 2 seconds. The numbers below are illustrative; yours come from your own measurements.

To turn 1.2 seconds into a number visible on a dashboard you need Little's law: the mean number waiting equals the arrival rate times the mean wait (L = λW). If a replica receives 20 requests per second, a mean wait of 1.2 seconds means 24 requests in the queue on average. The alert threshold on vllm:num_requests_waiting should sit well below 24: Little's law relates means, the promise is about the 95th percentile, and with bursty arrivals the waiting tail is far longer than the mean.

The same budget can be expressed in tokens. If a replica processes, say, 20 thousand input tokens per second, 1.2 seconds of queue hold no more than 24 thousand unprocessed input tokens. Current vLLM versions have a setting for exactly this: max_num_queued_tokens caps the total input of requests still waiting to be processed and returns 503 above the limit. The scheduler documentation calls it a TTFT quality-of-service mechanism and suggests setting it to the target TTFT times the input processing rate.

The tail compounds along a chain of calls

The last multiplier of the tail arrives with agents. The Tail at Scale (Dean and Barroso, CACM 2013) shows that if each server is slow one time in a hundred, a request waiting on a hundred servers is slow 63 % of the time. An agent spreads the same arithmetic over time: a session of 10 sequential model calls, each with a 1 % chance of crossing the threshold, crosses it at least once with probability 1 − 0.99¹⁰ ≈ 9.6 %; with 20 calls, about 18 %. A promise per call is not a promise per task, and agentic scenarios need it recalculated.

The second source of surprises is the unit of capacity. In an ordinary service it is intuitive: so many concurrent requests, so many threads. In inference, capacity is decided by whether the KV caches of all active requests fit side by side. The unit is memory for contexts, not open connections.

How expensive that memory is was shown by the PagedAttention paper (SOSP 2023): in the systems of the time only 20.4–38.2 % of KV-cache memory held actual token states, the rest went to reservations for the maximum length and to fragmentation. Paged memory layout raised throughput 2–4× at the same latency and became the basis of vLLM. The paper is covered in detail in Research Insights episode 32.

A worked example: how many conversations fit on a node

The KV-cache size per token follows from the model architecture: two tensors (keys and values) for every layer, for every KV head, times the head size, times bytes per number.

bytes per token = 2 × layers × KV heads × head size × bytes per number

Llama 3.1 70B, per the architecture description in Meta's repository, has 80 layers, 8 KV heads and a head size of 128 (8192 ÷ 64). In BF16 that is 2 × 80 × 8 × 128 × 2 = 327,680 bytes, or 320 KiB for every context token. Now take an illustrative node of four 80 GB H100 accelerators with tensor parallelism:

• —the engine gets 92 % of memory by default — the gpu-memory-utilization value in current vLLM documentation; of 320 GB, about 294 GB remain;
• —BF16 weights take about 140 GB: 70 billion parameters at 2 bytes each;
• —set aside a nominal 10 GB for activations and compute graphs;
• —about 144 GB remain for the KV cache, roughly 440 thousand tokens.

A conversation with 2.5 thousand input tokens and a 0.5 thousand token answer takes 3 thousand tokens, and about 145 such conversations fit on the node at once. One agent session with a 128 thousand token context takes 29 % of the pool — as much as 43 conversations. Switching the KV cache to FP8 halves the bytes per token and roughly doubles that room. The calculation is approximate: the engine allocates memory in blocks, shares common prefixes and reserves space for generation. Check it against the lines vLLM prints at startup: GPU KV cache size: N tokens and Maximum concurrency for X tokens per request. The documentation advises adding accelerators if these numbers fall short of what you need.

Hence a consequence that breaks intuition: one request with a very long context costs more than dozens of short conversations. And it is the one that evicts its neighbours. When memory runs short, vLLM logs an honest sentence: Sequence group N is preempted by PreemptionMode.RECOMPUTE mode because there is not enough KV cache space.

Preemption is a robustness mechanism rather than a failure: the system prefers to withdraw some work so the rest keeps moving. But it has a price. The preempted request is recomputed, and its end-to-end latency jumps. In V1 the default is recomputation rather than swapping out state — in that architecture recomputing is cheaper than moving memory around.

These are settings from the documentation, not a universal recipe. Choose them for the constraint in your service and test on the same workload. Dividing computations is tensor parallelism; separating processing stages is pipeline parallelism.

In SGLang the same knobs go by other names: --max-running-requests, --chunked-prefill-size and --schedule-conservativeness — the higher the value, the more cautiously the scheduler admits new requests. The SGLang tuning page is telling: it calls KV-cache usage above 0.9 good utilization and advises keeping the queue in the hundreds of requests. But that section is about maximum throughput for offline batch processing. For an interactive promise such a queue means seconds before the first token. The same metric is good or bad depending on what was promised.

Both engines support priorities. In vLLM it is --scheduling-policy priority: the lower value is served first. In SGLang it is --enable-priority-scheduling, where higher values go first by default and a separate flag flips the direction. The scales are opposite, and that belongs in the contract with clients — otherwise, after an engine change, a batch job ends up ahead of an interactive one.

The knob that switches between two promises

One parameter deserves separate billing: the cap on tokens per batch. V1 mixes input processing with generation in one batch by default, and this number allocates the machine between the two promises from the first section. Lower values improve the pace of generation, because input processing interrupts neighbours' generation less often. Higher values improve time to first token.

Where this trade-off comes from is shown by Sarathi-Serve (OSDI 2024). Schedulers that put input processing first cause generation stalls: in vLLM of that time such a stall could last several seconds. Splitting the input into chunks lets new requests join the batch without pausing the neighbours' generation, and under a limit on the 99th percentile of inter-token latency it delivered up to 2.6× more capacity for Mistral-7B on a single A100.

It is a rare case where the trade-off is not buried in a heuristic but exposed as a single configuration value. Which makes the decision a product one: which of the two thresholds you would rather breach.

If one knob is not enough, the next step is disaggregated inference: input processing and generation run on separate accelerator pools and scale independently. In DistServe that gave up to 7.4× more requests or a 12.6× tighter SLO with more than 90 % of requests inside the thresholds. In llm-d 1.0 of 20 October 2026 the pools of the two phases are managed as one logical unit for coordinated rollouts and upgrades. The price is moving the KV cache between pools and a second capacity model; where the proven gain ends is the subject of the LLM inference frontier.

Failures of an inference service collapse neatly into a small taxonomy. Its value is that every row has an early signal — one that appears before the first complaint.

Queues deserve a note of their own, because an old reliability rule applies. A queue is not free waiting: it consumes memory and adds latency. Google's book on running production systems gives a plain calculation — if a request takes 100 milliseconds and the queue is ten times the size of the worker pool, a request takes 1.1 seconds when the queue is full, almost all of it spent waiting. The recommendation is to keep queues short relative to the pool and reject early instead of accumulating.

In vLLM that advice has become a setting. Besides the unprocessed-input limit above, max_num_queued_reqs caps the number of requests running and waiting; above the limit the server returns 503 so the client can retry on another instance. Both limits are off by default, and without them the vLLM queue is unbounded: a short queue has to be switched on explicitly.

Early rejection is not just textbook advice. Mooncake, the serving platform of Moonshot AI's Kimi, is described in a paper that won the FAST 2025 best paper award. Its scheduler proactively rejects requests predicted to miss the SLO, to take load off the cluster. Overall, the architecture let Kimi handle 115 % and 107 % more requests on A800 and H800 clusters. In the extended version of the paper the authors call overload a common state at peak times and report a separate experiment: on a recording of 23 thousand real requests replayed at double speed, the baseline rejected 4,183 requests, early rejection 3,771, and early rejection with load prediction 3,589. Fewer requests are rejected because the cluster stops spending work on requests that would miss the deadline anyway.

Retries deserve their own paragraph, because in inference they cost more. In the Handling Overload chapter of Google's book, even a limit of three attempts per request nearly triples the load under widespread failures. So a second budget is added: a client retries only while retries make up less than 10 % of its requests, which in the general case caps the growth at about 1.1×. The same chapter describes client-side adaptive throttling: the client itself drops a share of requests with probability max(0, (requests − K × accepts) / (requests + 1)) over the last two minutes, and the authors prefer K = 2. The neighbouring chapter adds a server-side budget — for example, no more than sixty retries a minute per process.

What is specific to inference is that a retry does not merely add a request to the queue. It claims memory for the whole context again — the very resource whose exhaustion caused the rejection. An unbudgeted retry here does not just lengthen the queue: it evicts requests that were nearly finished and turns degradation into failure.

An ordinary service has a poor choice under overload: serve or refuse. A generative service has a third option, and it is almost always better — return a cheaper answer. Between "as usual" and "error" lies a whole scale.

• —cap the answer length: shorter, but on time;
• —route the request to a smaller model and say so in the interface;
• —switch off the expensive reasoning mode where it is not critical;
• —narrow the document search: fewer retrieved passages, a faster answer;
• —move the request to a deferred lane and return the result later.

Keep the difference between load shedding and degradation explicit: the first drops traffic so the server survives, the second lowers the quality of the work instead of failing. Both share one problem: this code runs rarely, so by the time a real incident arrives it has most likely rotted. The degradation path has to be exercised regularly — otherwise it exists only on a diagram.

The second instrument is priority. An interactive request and a batch job should not share one queue as equals, and that is now recognised at the level of platform standards. The Gateway API Inference Extension for Kubernetes exists precisely because ordinary load balancing does not fit here: sessions are long, resource-intensive and partially stateful, and a single pod keeps several active sessions together with their token caches in memory. Routing by HTTP path or round-robin cannot see any of that.

The project introduces a pool of model servers owned by the platform, and a separate component that picks an endpoint against configured objectives — prefix-cache state and adapter availability are among the signals it names. The announcement is dated 5 June 2025, and the model server pool became a stable API in version 1.0 in September 2025. Within a year the boundary moved: in version 1.6 (August 2026) the full endpoint-picking component and the priority objects moved to llm-d, and the project itself narrowed to the standard, the Kubernetes resources and conformance tests.

In llm-d priority and load shedding live in one mechanism. A request's objective sets its priority; a negative value marks the request as sheddable. Flow control places requests into per-priority queues, caps the capacity of each and returns 429 when a limit is exhausted; if needed, it can end an already dispatched low-priority request in favour of a more important one. llm-d 1.0 presents this as priority- and fairness-aware flow control, but the documentation still puts the mechanism behind a separate setting.

There is a third path between preemption and rejection: move the request. Llumnix (OSDI 2024) from Alibaba migrates a request together with its KV cache to another instance with near-zero downtime and uses that for load balancing, priorities and draining nodes. In the authors' experiments the tail latency improved by an order of magnitude, high-priority requests got up to 1.5× faster, and costs fell by up to 36 % at a comparable tail. The mechanism is covered in the Llumnix review from the series on cluster inference.

Queue management as a problem of its own is described in QLM (SoCC 2024): the system keeps interactive and batch requests with different SLOs in one queue and orders it by an estimate of each request's waiting time; the same estimate drives eviction, load balancing and model swapping. The evaluation rests on a cloud provider's production requirements. The authors report SLO attainment up 40–90 % and throughput up 20–400 %. The SLOs-Serve preprint goes further and allocates tokens under multiple SLO constraints: it reports 2.2× average per-accelerator serving capacity across six scenarios, from summarization to tool calling. As of 31 October 2026 the preprint still has a single April 2025 version and no confirmed venue, so it is a signal of direction rather than an expected gain.

Autoscaling inference starts with picking a metric, and the first temptation is the worst one. Accelerator utilization is a poor signal: it does not measure how much useful work happens while the accelerator is busy, so it correlates badly with both latency and throughput.

Two signals do work, and they answer different questions. Queue size shows that requests are already waiting: the guidance is to start with a threshold around three to five and raise it until latency reaches the target. Batch size shows how much the system is processing right now and suits low-latency targets better; its threshold is found empirically — observe the maximum under load, set slightly below it, and lower until latency is where you want it. This is also where it becomes clear why a larger batch raises throughput and latency at once: the input processing of some requests interrupts the generation of others.

Platforms reduce these signals to SLO targets. The NVIDIA Dynamo planner in version 1.6 changes the number of input-processing and generation replicas to hold targets for time to first token and inter-token latency; in its default mode it relies on thresholds for queue depth and KV-cache usage. In llm-d's autoscaling guide one option converts the backlog of unprocessed input into seconds of waiting and compares them with a share of the first-token budget — the same calculation as in the worked example above. The same guide notes that during active batching accelerator utilization sits near 100 % regardless of load, so it is useless as a signal. The signal is the same everywhere: the queue and memory.

The rule fired — and nothing happened for several more minutes

Then comes the part that separates operating inference from operating a web service. Scaling means a new pod, and a new pod pays a full cold start: pull the image, initialise the runtime, load gigabytes of weights, compile the compute graphs. Minutes pass before it does useful work — and the queue that triggered the rule has already produced its tail.

Research attacks this interval from two sides. ServerlessLLM (OSDI 2024) keeps weights on local disks and in node memory, loads them 3.6–8.2× faster than standard formats and cuts latency by 10–200× depending on the workload; details are in the cold-start review. BlitzScale (OSDI 2025) loads weights over the compute network between accelerators and offloads the computation of individual layers to new instances without waiting for the full load: the authors report up to 94 % lower tail latency than ServerlessLLM and 49 % less accelerator time.

Open-source tools shorten the same interval by waking instead of loading. llm-d's fast model actuation guide describes an approach where tensors stay in host memory and return to the accelerator in seconds, while starting a new instance with modules already loaded takes tens of seconds. The project is explicit that this accelerates how fast capacity arrives, not inference itself, and puts the overhead of resident servers at about 2.5 percent of host memory.

vLLM's numbers for the same mode: on an A100, waking takes 0.26 seconds for Qwen3-0.6B and 0.82 seconds for Phi-3-vision against a 37–58 second cold start. On an A100 model switching speeds up 18–20×, while the upper end of the claimed range — about 200× — was measured on an A4000. These are the project's measurements on its own models and hardware — a vendor claim, not a result to expect on your own stack.

The practical conclusion of this section is not about tooling. While the reaction to load is measured in minutes, headroom is not waste but the price of physics. The same plot has already played out at the level of the platform itself: complexity does not disappear, it relocates — out of the application into the platform, and out of the platform onto whoever operates it. A separate piece on Kubernetes describes that mechanism for internal developer platforms.

All of the above reduces to one management choice. Headroom costs money every hour, whether or not it was needed. A breached promise costs money at the moment of the breach — and that cost is unevenly distributed: for an internal tool it is close to irritation, for a product path it is measured in abandoned sessions.

For the same period: headroom cost = hourly price × hours; breach losses = request count × share of breached requests × average cost per breach

Compare costs for the same period and workload. Estimate losses for each scenario using its own breach rate; extra headroom is worthwhile if the expected reduction in losses exceeds its cost.

The temptation is to count only the left side, because it arrives as an invoice. The right side needs somebody to name the price of a broken promise, and that is not an engineering conversation. Without it, every capacity discussion ends the same way: "expensive" beats "slow" until "slow" becomes visible from outside.

A worked example: the break-even point of headroom

Naming the price of a breach is hard, so it helps to turn the question around and compute the price at which headroom pays off. All numbers below are illustrative, and prices are in nominal units.

• —an interactive scenario: 1.2 million requests a day, with bursts during working hours;
• —without headroom autoscaling catches up with each burst after several minutes of cold start, and 25,000 requests a day miss the first-token threshold;
• —one extra replica running around the clock costs 40 units an hour, or 960 units a day, and removes 90 % of breaches — 22,500 requests.

breach price at which headroom pays off = 960 ÷ 22,500 ≈ 0.043 units

Now the conversation with the product comes down to one question: is a single broken promise worth more than four hundredths of a unit? For a paid coding assistant the answer is almost certainly yes; for an internal tool, perhaps not. The decision has intermediate forms too. If bursts fall within eight working hours, scheduled headroom costs 320 units a day, and the break-even drops to 0.014 units per breach. If waking a sleeping replica cuts the cold start to seconds, the headroom needed shrinks further: the window it covers gets shorter.

It helps to count accepted work rather than tokens — the same unit used in the piece on the economics of AI development. Then headroom stops being a cost line and becomes a multiplier: it raises the share of requests that reach an accepted result.

An honest boundary for this section. The calculation shows the shape of the decision, not its outcome. Someone else's numbers cannot be substituted into it: the hourly price depends on your contract, the breach rate on the traffic profile and how fast replicas come up, and the price of a breach on the scenario. Recompute the break-even point on your own measurements whenever any of the three changes.

Everything in one place. If one page has to survive the article, this is the one.

• —Write the promise down as a pair of thresholds plus the share of requests inside them — separately for interactive and batch paths. For agentic scenarios recalculate it per task, not per call.
• —Split the three numbers on the dashboard: time to first token, pace of generation, end-to-end latency. Do not average them into one curve.
• —Give queue time its own metric: it is the boundary between what the model owns and what capacity owns.
• —Split the first-token threshold into a budget and derive, from what is left for the queue, an alert threshold on waiting requests and a limit on unprocessed input.
• —Compute capacity in KV-cache tokens from the model architecture and check it against the cache-size line at engine startup; plan by the distribution of context length, not by user count.
• —Watch KV-cache occupancy and preemption counts as early signals; they move before latency does.
• —Admit less at the door: a short queue with an early rejection beats a long queue with a late timeout. In vLLM the queue limits are off by default.
• —Tag requests with priorities and check the direction of the scale in your engine and router.
• —Define degradation up front — length, model, mode, retrieval depth — and exercise that path on a schedule.
• —Add a retry budget and exponential backoff: in inference a retry claims memory for the whole context again.
• —Scale on queue size or batch size, never on accelerator utilization.
• —Budget for the replica cold start: the rule fires instantly, capacity arrives minutes later.
• —Compute the break-even point of headroom and revisit it when the traffic profile or replica start-up speed changes.