Operating LLM Inference: What to Promise, How to Measure Capacity, and How to Degrade

The same inference service can be described two honest ways. The engineer shows a curve: so many tokens per second at saturation, ahead of the competition, twice as fast as last quarter. The product shows something else: the share of conversations where the answer took more than three seconds to start has doubled. Both measurements are correct, and they barely relate to each other.

The difference is not measurement rigour but what is being measured. Throughput describes how much work the machine completes when it is fully loaded. A product does not live in the average; it lives in the tail — in what happened to the requests unlucky enough to queue behind somebody's very long context. A benchmark without a threshold describes hardware. A promise describes behaviour at the boundary.

You do not have to invent the format of that promise. The industry has already written it down. MLPerf Inference v5.1 rates a system on Llama 3.1-8B not by raw speed but by the load it sustains inside two bounds. In the Server scenario that means TTFT at or below 2 seconds and TPOT at or below 100 milliseconds. In the newer Interactive scenario the bounds tighten to 0.5 seconds and 30 milliseconds.

MLCommons translates those thresholds into a human unit itself: 100 milliseconds per token is roughly 480 words per minute, noticeably faster than people read, while 30 milliseconds is about 1,600 words per minute. The second scenario exists for cases where responsiveness is the point: coding assistants and real-time creative tools.

promise = a threshold on the start of the answer + a threshold on the pace of generation + the share of requests that stay inside both

Everything that follows is about where the breach of that promise comes from: why one latency curve is not enough, where the tail actually lives, what capacity is measured in, how the whole thing breaks, and what to do instead of failing.

For a generative service, "slow" splits into at least three different events. The user presses a button and nothing happens — that is time to first token. The answer starts but prints slower than it reads — that is the pace of generation, the time per output token. The whole answer does not arrive before the caller times out — that is end-to-end latency.

Averaging these into a single curve loses information rather than simplifying it. Time to first token grows with the queue and with input length; the pace of generation drops when the machine gains neighbours; end-to-end latency can grow simply because answers got longer. Three causes, three knobs.

The good news is that the engine has already separated them. vLLM exposes histograms for vllm:time_to_first_token_seconds, vllm:inter_token_latency_seconds and vllm:e2e_request_latency_seconds, and next to them something an ordinary service does not have: vllm:request_queue_time_seconds, vllm:num_requests_running and vllm:num_requests_waiting.

A dedicated queue-time metric is not a detail. It separates what the model is responsible for from what the scheduler and the capacity are responsible for. If your dashboard shows one latency curve, that is a team decision, not a tooling limit — the split has already been made for you.

Engineering intuition says latency grows when capacity runs short. That is half true, and the other half explains most of the unpleasant incidents.

The classic approximation for mean waiting time is Kingman's formula, published in 1961. It factors waiting into three multiplicands: utilization, variability and service time.

wait ≈ [ρ / (1 − ρ)] × [(c²arrival + c²service) / 2] × mean service time

The managerial reading is that waiting grows for three independent reasons: arrivals become burstier, service times spread out, utilization approaches one. Buying capacity only addresses the third.

In inference the first two are extreme, and that is what separates it from an ordinary service. The same endpoint serves a request with a few hundred context tokens and one with hundreds of thousands; answers run from one line to ten screens. The spread of service time is measured in orders of magnitude, not percentages. That is why a cluster "only 60 percent loaded" confidently produces an unacceptable tail — and why an extra accelerator barely moves it.

Boundary of applicability. Kingman's formula approximates a single-server queue and is most accurate near saturation. An inference server is not a single-server queue: it batches requests, preempts them and reschedules on every iteration. Here the formula is a thinking tool, not a capacity calculator.

The second source of surprises is the unit of capacity. In an ordinary service it is intuitive: so many concurrent requests, so many threads. In inference, capacity is decided by whether the KV caches of all active requests fit side by side. The unit is memory for contexts, not open connections.

Hence a consequence that breaks intuition: one request with a very long context can cost more than a dozen short conversations. And it is the one that evicts its neighbours. When memory runs short, vLLM logs an honest sentence: Sequence group N is preempted by PreemptionMode.RECOMPUTE mode because there is not enough KV cache space.

Preemption is a robustness mechanism rather than a failure: the system prefers to withdraw some work so the rest keeps moving. But it has a price. The preempted request is recomputed, and its end-to-end latency jumps. In V1 the default is recomputation rather than swapping out state — in that architecture recomputing is cheaper than moving memory around.

The documentation also lists the knobs: raise the share of memory given to the engine, lower the cap on concurrent sequences or on tokens per batch, spread the model across more accelerators along the tensor or pipeline dimension. Each pushes the same boundary from a different side.

The knob that switches between two promises

One parameter deserves separate billing: the cap on tokens per batch. V1 mixes input processing with generation in one batch by default, and this number allocates the machine between the two promises from the first section. Lower values improve the pace of generation, because input processing interrupts neighbours' generation less often. Higher values improve time to first token.

It is a rare case where the trade-off is not buried in a heuristic but exposed as a single configuration value. Which makes the decision a product one: which of the two thresholds you would rather breach.

Failures of an inference service collapse neatly into a small taxonomy. Its value is that every row has an early signal — one that appears before the first complaint.

Queues deserve a note of their own, because an old reliability rule applies. A queue is not free waiting: it consumes memory and adds latency. Google's book on running production systems gives a plain calculation — if a request takes 100 milliseconds and the queue is ten times the size of the worker pool, a request waits a full second before any work starts. The recommendation is to keep queues short relative to the pool and reject early instead of accumulating.

Retries deserve their own paragraph, because in inference they cost more. The classic amplification arithmetic: if a hundred requests per second are rejected under overload and clients retry once a second, the load becomes 200, then 300 requests per second. The remedy is randomized exponential backoff plus a server-side retry budget — for example, no more than sixty retries a minute.

What is specific to inference is that a retry does not merely add a request to the queue. It claims memory for the whole context again — the very resource whose exhaustion caused the rejection. An unbudgeted retry here does not just lengthen the queue: it evicts requests that were nearly finished and turns degradation into failure.

An ordinary service has a poor choice under overload: serve or refuse. A generative service has a third option, and it is almost always better — return a cheaper answer. Between "as usual" and "error" lies a whole scale.

• —cap the answer length: shorter, but on time;
• —route the request to a smaller model and say so in the interface;
• —switch off the expensive reasoning mode where it is not critical;
• —narrow the document search: fewer retrieved passages, a faster answer;
• —move the request to a deferred lane and return the result later.

Keep the difference between load shedding and degradation explicit: the first drops traffic so the server survives, the second lowers the quality of the work instead of failing. Both share one problem: this code runs rarely, so by the time a real incident arrives it has most likely rotted. The degradation path has to be exercised regularly — otherwise it exists only on a diagram.

The second instrument is priority. An interactive request and a batch job should not share one queue as equals, and that is now recognised at the level of platform standards. The Gateway API Inference Extension for Kubernetes exists precisely because ordinary load balancing does not fit here: sessions are long, resource-intensive and partially stateful, and a single pod keeps several active sessions together with their token caches in memory. Routing by HTTP path or round-robin cannot see any of that.

The project introduces a pool of model servers owned by the platform, and a separate component that picks an endpoint against configured objectives — prefix-cache state and adapter availability are among the signals it names. The announcement is dated 5 June 2025; as of September 2026 the project ships a v1 API alongside the alpha.

Research pushes the same idea further: the SLOs-Serve preprint allocates tokens under multiple SLO constraints and reports 2.2× average per-GPU serving capacity across six scenarios, from summarization to tool calling. Treat it as a signal of direction rather than an expected gain: the result comes from the authors' own scenario set and has not been independently reproduced.

Autoscaling inference starts with picking a metric, and the first temptation is the worst one. Accelerator utilization is a poor signal: it does not measure how much useful work happens while the accelerator is busy, so it correlates badly with both latency and throughput.

Two signals do work, and they answer different questions. Queue size shows that requests are already waiting: the guidance is to start with a threshold around three to five and raise it until latency reaches the target. Batch size shows how much the system is processing right now and suits low-latency targets better; its threshold is found empirically — observe the maximum under load, set slightly below it, and lower until latency is where you want it. This is also where it becomes clear why a larger batch raises throughput and latency at once: the input processing of some requests interrupts the generation of others.

The rule fired — and nothing happened for several more minutes

Then comes the part that separates operating inference from operating a web service. Scaling means a new pod, and a new pod pays a full cold start: pull the image, initialise the runtime, load gigabytes of weights, compile the compute graphs. Minutes pass before it does useful work — and the queue that triggered the rule has already produced its tail.

Tools against this exist. The llm-d project describes an approach where the model is not reloaded but woken: tensors stay in host memory and return to the accelerator in seconds, skipping loading and compilation. The project is explicit that this accelerates how fast capacity arrives, not inference itself, and puts the overhead of resident servers at about 2.5 percent of host memory.

vLLM's numbers for the same mode: on an A100, waking takes 0.26 seconds for Qwen3-0.6B and 0.82 seconds for Phi-3-vision against a 37–58 second cold start, with a claimed 18× to 200× range for model switching. These are the project's measurements on its own models and hardware — a vendor claim, not a result to expect on your own stack.

The practical conclusion of this section is not about tooling. While the reaction to load is measured in minutes, headroom is not waste but the price of physics. The same plot has already played out at the level of the platform itself: complexity does not disappear, it relocates — out of the application into the platform, and out of the platform onto whoever operates it. A separate piece on Kubernetes describes that mechanism for internal developer platforms.

All of the above reduces to one management choice. Headroom costs money every hour, whether or not it was needed. A breached promise costs money at the moment of the breach — and that cost is unevenly distributed: for an internal tool it is close to irritation, for a product path it is measured in abandoned sessions.

decision = (price of an hour of headroom × hours) against (share of breached requests × price of one breach)

The temptation is to count only the left side, because it arrives as an invoice. The right side needs somebody to name the price of a broken promise, and that is not an engineering conversation. Without it, every capacity discussion ends the same way: "expensive" beats "slow" until "slow" becomes visible from outside.

It helps to count accepted work rather than tokens — the same unit used in the piece on the economics of AI development. Then headroom stops being a cost line and becomes a multiplier: it raises the share of requests that reach an accepted result.

An honest boundary for this section. This is a decision frame, not a calculation. Someone else's numbers cannot be substituted into it: the hourly price depends on your contract and the price of a breach on your scenario. The section stays a frame until it carries measurements of its own.

Everything in one place. If one page has to survive the article, this is the one.

• —Write the promise down as a pair of thresholds plus the share of requests inside them — separately for interactive and batch paths.
• —Split the three numbers on the dashboard: time to first token, pace of generation, end-to-end latency. Do not average them into one curve.
• —Give queue time its own metric: it is the boundary between what the model owns and what capacity owns.
• —Watch KV-cache occupancy and preemption counts as early signals; they move before latency does.
• —Admit less at the door: a short queue with an early rejection beats a long queue with a late timeout.
• —Define degradation up front — length, model, mode, retrieval depth — and exercise that path on a schedule.
• —Add a retry budget and exponential backoff: in inference a retry claims memory for the whole context again.
• —Scale on queue size or batch size, never on accelerator utilization.
• —Budget for the replica cold start: the rule fires instantly, capacity arrives minutes later.
• —Revisit the headroom decision when the traffic profile changes: context length moves capacity more than user count does.