The LLM Inference Frontier: Ten Unproven Directions That Promise Multiplicative Gains

The previous article was about the norm: continuous batching, paged KV memory, prefix caching, chunked prefill, phase disaggregation. The norm has a convenient property — you can measure it on your own trace and buy it ready-made in an open engine. The frontier has different properties. Its results were obtained on other people's models and accelerators and, as a rule, against a baseline convenient to the authors. So before listing directions, we need to agree on how to read claims about them.010101I keep coming back to a point by Dylan Patel that I covered in the channel: a doubling in the model, a doubling in the kernels, and a doubling in the hardware do not add up to eight times when optimized separately, but give an order-of-magnitude jump when designed together. Almost everything below is an attempt to co-design what used to be optimized in isolation.Knizhny kub · Dylan Patel on co-design01I keep coming back to a point by Dylan Patel that I covered in the channel: a doubling in the model, a doubling in the kernels, and a doubling in the hardware do not add up to eight times when optimized separately, but give an order-of-magnitude jump when designed together. Almost everything below is an attempt to co-design what used to be optimized in isolation.Knizhny kub · Dylan Patel on co-design

A six-step ladder helps. An idea exists as a preprint or a demo. Another team has reproduced the effect or measured its limits. The feature has appeared in an open engine — vLLM, SGLang, or TensorRT-LLM. There is a public production report from one company. There are several companies with different workloads. Finally, the technique has become the default expectation, and its absence needs an explanation. PagedAttention climbed all six steps in two years. Phase disaggregation in August 2026 sits between the third and the fourth: vLLM's documentation still marks the feature experimental, while Moonshot and DeepSeek run it in production. Nobody jumps from the first step straight to the fifth, and each step is a separate proof.

Three questions apply to any claim. First: which resource is saved — bytes read from memory, serial steps, accelerator idle time, or watts. A technique that saves a resource other than the one limiting your workload yields zero. Second: against which baseline. 'Ten times faster' against an engine with no cache and '1.2 times' against a tuned vLLM may describe the same system. Third: what must become true for the gain to happen for you — kernels for the format, hardware of the right generation, model quality on your tasks.

A separate trap is tokens per second per user versus goodput per accelerator. Speculative decoding shows it best. EAGLE 3.1 on Kimi K2.6 (GB200, May 2026) gives 2.03× at one stream and 1.66× at sixteen; P-EAGLE versus EAGLE-3 gives 1.55–1.69× at one stream and only 1.05–1.25× at 64. Almost every 'N×' on the frontier was obtained at low concurrency, and a platform team lives at high concurrency.

The ten directions below run from the model to the data center: how the model computes (architectures and parallel generation), how the cluster cuts the work (disaggregation below the phase), where state lives (KV cache as storage), in which numbers the weights are stored (FP4 and below), on which silicon (beyond the GPU), which workloads all of it serves (reasoning, agents, multimodality), and what limits it from above (power, control, trust). The last section collects them into a single maturity map.

The entire serving stack of the previous article is built around one property of the transformer: per-request state grows by one KV block with every token and is read in full at every decode step. Paged memory, prefix caching, KV transfer between pools — all are tools for a growing object. The 2025–2026 frontier attacks the property itself in two ways: make the state fixed-size, or stop reading all of it.

The first way is linear attention and state-space models, where a request owns a fixed-size state that is overwritten rather than accumulated. The working layer of the hybrids became Gated DeltaNet (NVIDIA, December 2024): a forget gate plus the delta rule. It underlies Qwen3-Next (September 2025: three DeltaNet layers per gated-attention layer), Qwen3.5-397B-A17B (February 2026, the same recipe with a one-million-token context), Kimi Linear (October 2025, the KDA variant, 3:1 with MLA), and Kimi K3 (July 2026: 93 layers, 69 of them KDA and 24 MLA, 2.8 trillion parameters with roughly 104 billion active). NVIDIA takes the same road through Mamba-2: Nemotron-H (April 2025), Nemotron 3 Nano (December 2025: 23 Mamba-2 layers, 6 attention, 23 MoE), and Nemotron 3 Ultra (June 2026, 550B/55B). IBM Granite 4.0 keeps a 9:1 ratio; Falcon-H1 puts both kinds of heads in one layer. Mamba-3 (March 2026) and Gated DeltaNet-2 (May 2026) remain preprints.

The only controlled same-size comparison came from Kimi Linear: 48B with 3B active against the same model on MLA — KV cache 75% smaller, time per output token 6.3× better at a one-million context, 2.2× at 512K, and a negligible gain at 128K; RULER-128K 84.3 versus 81.3. The other multipliers are vendor-supplied and composite. Qwen3-Next's 'ten times the throughput above 32K context' is measured against Qwen3-32B and mixes MoE, multi-token prediction, and the attention architecture itself. Nemotron 3 Nano's 'up to 3.3×' against Qwen3-30B-A3B was obtained on one H200 at 8K input and 16K output in FP8. Nemotron 3 Ultra's 'about 6×' against GLM-5.1 and Kimi K2.6 turns into 1.6× against Qwen3.5, itself a hybrid — and with different serving stacks. Long context is ambiguous: Qwen3.5 reaches RULER 80.3 at one million versus 84.5 for Qwen3-235B with full attention — the hybrid wins up to 256K and trails at 512K and beyond.

The second way is sparse attention: the state grows, but only a selected subset is read at each step. NSA (February 2025) showed 11.6× on decode at 64K versus FlashAttention-2 on A100; Moonshot's MoBA serves Kimi's long context. DeepSeek Sparse Attention in V3.2-Exp (29 September 2025) introduced a lightweight indexer that selects 2,048 tokens under MLA — O(L·k) instead of O(L²), quality 'on par with V3.1-Terminus', and the API price cut by more than half the same day; GLM-5 adopted DSA (February 2026). DeepSeek-V4 (April 2026) went further: compressed sparse and heavily compressed attention together give, at one million tokens, 27% of the FLOPs and 10% of the KV cache of V3.2, and an on-disk KV cache became a regular storage tier. This line has its own bill: by LongCat's measurements (Meituan, August 2026) the DSA indexer takes up to 90% of per-layer latency at one million, and a hierarchical index speeds training by half again while decode improves by only 4–13%.

The systems layer trails the models by about a year

The engines described what breaks themselves. Per vLLM (November 2025 and April 2026), a 16-token KV block takes about 64 KiB while a Mamba state takes about 2.57 MiB per sequence; at 128K context the KV cache is roughly 200 times larger than the state. The memory win is obvious, but the state is overwritten in place and cannot be 'rewound'. Prefix caching loses partial matches: vLLM inflates the attention block to 528–672 tokens so that it lines up with the state page, and in April 2026 that surfaced as issue #40696 — prompts shorter than 528 tokens got roughly zero hits, and production QPS fell from 200 to under 100 once prompts crossed below the boundary. SGLang (December 2025) went through checkpoints: the prefix tree stores state snapshots, speculation gets a private state slot per draft token, and phase disaggregation ships the state as one contiguous block rather than a stream of pages. Marconi (MLSys 2025) showed that the capacity of such a cache must be counted in FLOPs, not tokens.

Moonshot disclosed the price of checkpoints for K3: about 0.4 GiB each, roughly 40 GB per one-million context at a 10K-token interval. The company contributed KDA prefix caching to vLLM itself and states that its API runs disaggregated with a cache hit rate 'above 90%' on coding — a vendor claim. vLLM shipped phase disaggregation for hybrid models in April 2026: the state travels over NIXL as one block, and on 8×H200 the disaggregated setup beats the co-located one above roughly 64 concurrent users; for DeltaNet models the feature is still 'pending'. The ReplaySSM proposal (July 2026) — buffer state inputs instead of writing the state every step — gives 1.40× and 1.21× at batch 128 but was not merged as of 22 August.

The counter-argument from those who tried and went back

MiniMax-M1 (June 2025) was built on lightning attention at a 7:1 ratio. In a note dated 29 October 2025, the head of pretraining explained why M2 stayed a full-attention model: hybrids matched it on MMLU, BBH, MATH, and LongBench but at scale showed 'clear deficits in complex, multi-hop reasoning'; linear attention is 'far more sensitive to numerical precision', which sits badly with low-precision state storage; coexistence with prefix caching, which has very high hit rates in dialogue, and with speculative decoding is 'an open and unsolved problem'; the infrastructure is immature, with many linear kernels memory-bound even in training. M2.7 (May 2026) is still full attention in every layer, and the same team's sliding-window hybrid ablation scores RULER-128K 72 versus 90.

Research papers add boundaries. Meta FAIR (October 2025) found the optimum near one attention layer per five Mamba layers but showed that hybrids keep needle-in-a-haystack retrieval only to about 1.5× the training length. HALO (January 2026) recorded distilled hybrids collapsing at 128K: Jet-Nemotron 2B scores 0.0 against 96.4 for Qwen3. The Mooncake authors showed the other side of the coin in April 2026: hybrid attention shrinks KV enough that prompt processing can be moved to another data center — +54% throughput and −64% P90 TTFT on an internal trillion-parameter model. The transfer object changes, and with it the geography of inference.

The second way to change decode physics is to leave the state alone and reduce the number of serial steps. Three branches: diffusion language models that produce a block of tokens over several passes, multi-token prediction inside the model itself, and next-generation speculation, where a drafter writes a whole block in one forward pass.

A diffusion language model denoises a canvas of hundreds of tokens over dozens of steps, committing at each step to the positions it is confident about. Mercury from Inception (June 2025) showed 1,109 tokens per second per user on an H100 for Mercury Coder Mini; Mercury 2 (24 February 2026) became the first 'reasoning' diffusion model with a 128K context — the vendor claims 1,009 tokens per second on Blackwell, while Artificial Analysis measured 881, a first token after 3.62 seconds, and 22 points on its intelligence index (63rd of 172). Gemini Diffusion remains an 'experimental demo' fifteen months after it was shown. The most candid document here is DiffusionGemma (10 June 2026): Gemma 4 at 26B with 4B active plus a diffusion head, a 256-token canvas, up to 48 steps, 15–20 tokens per forward, 1,008 tokens per second on an H100 in FP8 at batch 1 — '5–6 times faster than autoregression'. Quality is below the original Gemma 4: MMLU Pro 77.6 versus 82.6, AIME 2026 69.1 versus 88.3, GPQA Diamond 73.2 versus 82.3. And Google's own caveat: at high QPS autoregressive models saturate the compute, parallel decoding offers diminishing returns 'and can result in higher serving costs'; the speedup is designed for local and low-concurrency inference.

The 100B scale belongs to LLaDA2.0 (Ant Group, December 2025): an MoE converted from an autoregressive model reaches 500 tokens per second against 258 for the baseline on 8×H20 — 'up to 1.9×' at low concurrency; SGLang's roadmap for diffusion models has one item closed out of eleven. ByteDance's Seed Diffusion Preview (July 2025) claims 2,146 tokens per second on an H20 and '5.4×' against an unnamed autoregressive model of the same scale. Serving such models is hard for structural reasons: bidirectional attention removes the exact KV cache, each canvas needs dozens of passes, requests converge at different rates so a batch waits for stragglers, and logits are materialized for the whole canvas. The 2025–2026 answers are block diffusion with KV across blocks, Fast-dLLM's approximate cache (27.6× — against a baseline with no cache; version 2 — 2.5× over autoregression), the dInfer engine (over 1,100 tokens per second on 8×H800 at batch 1 — against a 3B model on vLLM), and SGLang's reuse of chunked prefill. Optimus (May 2026) reaches the same conclusion as Google: 'throughput gains vanish beyond saturation'.

NVIDIA was the most candid of all. Nemotron-Labs-Diffusion (July 2026) is one model in three modes: autoregression, block diffusion, and self-speculation, where diffusion proposes and the autoregressive head verifies in one pass. In self-speculation the model yields 5.99 tokens per forward, in pure diffusion 2.57, and for high concurrency the authors recommend the autoregressive mode. I-DLM (April 2026) calls itself the first diffusion model to match its same-size autoregressive counterpart — at small scale. So diffusion turns from a replacement into a drafter.

Next-generation drafters and where they stop paying

Multi-token prediction is built into the model: DeepSeek-V3 accepts the second token 85–90% of the time and gets 1.8× at low load, but in SGLang on H200 the gain falls from 60.8% at two requests per rank to 14.2% at 128. MiMo-V2-Flash (Xiaomi, January 2026) claims 2.6× with three such layers; Qwen3-Next ships the module without published numbers. External drafters went further. EAGLE-3 (March 2025) became the engines' standard: up to 6.5× single-stream and 1.38× at batch 64. DFlash (February 2026, ICML) replaced the drafter with a five-layer block-diffusion network that emits a 16-token block in one forward: on Qwen3-8B 4.86× against 1.76–2.02× for EAGLE-3, and on B200 5.1× at four streams and 2.8× at 32. In June 2026 NVIDIA showed 'over 15×' against autoregression and 1.5× against EAGLE-3 on gpt-oss-120b and 8×DGX B300 — under a 500–600 tokens per second per user constraint, that is, as throughput under an interactive SLO, not as a peak. DBLAST (August 2026) added the caveat that accepted length degrades as the sampling entropy rises.

DeepSeek pulled it together in DSpark (July 2026): a parallel drafter with a light sequential correction head and a confidence-scheduled verification length. On V4-Pro and 8×B300 the first token of a seven-token block is accepted over 70% of the time and the seventh under 10%; per user it comes out 60–85% faster at matched throughput against one-step prediction, while 'at batch 256 draft tokens compete with real tokens for the same compute'. The method is 'deployed within the DeepSeek-V4 serving system'; on 14 August 2026 vLLM added adaptive verification. AngelSpec at Tencent and P-EAGLE at Amazon with NVIDIA are the same wave.

The first evaluation in a production-grade vLLM (December 2025) explained why the table looks the way it does: there is 'a substantial gap' between observed and theoretical speedups, verification dominates, and acceptance drifts by position, request, and dataset. The rule is simple: speculation pays while the verifying model is memory-bound; at 64–256 streams on Blackwell it is already compute-bound. The exception is long context, where verification hits KV reads even at large batch: MagicDec gets 2.51× at batch 32–256 with a sparse-KV drafter. ATLAS at Together (October 2025) showed one more lever — an adaptive drafter whose acceptance on a drifting RL workload rose from under 10% to over 80% within 1,400 steps.

Prefill/decode disaggregation cuts a request in time. The next cut runs inside a decode step, by layer type. Attention reads a request's entire KV cache for a handful of operations, and its arithmetic intensity is set by the architecture: about 32 operations per byte for GQA, 128 for MFA, 512 for MLA at 8-bit KV — and batching does not raise it, because KV is not shared between requests. FFN and expert weights, by contrast, are shared by every token in the step, and their intensity grows with tokens per expert until it hits compute. In a co-located pool one accelerator must hold enough KV to batch the FFN into its compute regime — capacity and bandwidth fight each other. Attention–FFN disaggregation sizes the attention pool by memory capacity and bandwidth and the expert pool by FLOPs, and hides the per-layer activation exchange behind alternating micro-batches.

Here m is the number of micro-batches in flight, T_comm the exchange latency between pools, and T_ffn the time of an expert layer. For Step-3, at a 50-millisecond budget per token and 61 layers, that is under 272 microseconds per layer including two network hops. MoE already pays an all-to-all per layer, so the extra hop is nearly free for it; for a dense model it is not, as the DistServe authors wrote plainly in their November 2025 retrospective: phase disaggregation became 'the industry standard', and attention–FFN disaggregation 'the next frontier' — for MoE only.020202I covered in the channel how Google and Meta are rebuilding their networks: both mottos boil down to the continent becoming the data center and every site becoming one computer. For splitting attention from experts, the network stops being background: it enters every layer's budget, and its tail latency decides how many micro-batches have to stay in flight.Knizhny kub · Google and Meta networks in the AI era02I covered in the channel how Google and Meta are rebuilding their networks: both mottos boil down to the continent becoming the data center and every site becoming one computer. For splitting attention from experts, the network stops being background: it enters every layer's budget, and its tail latency decides how many micro-batches have to stay in flight.Knizhny kub · Google and Meta networks in the AI era

There are three production measurements. MegaScale-Infer at ByteDance (SIGCOMM 2025): attention replicas feed an expert pool through a custom exchange library in place of NCCL — from 2.56× on Mixtral 8×22B to 7.11× on a 317B model in per-GPU decode over vLLM, and 1.28–1.90× over TensorRT-LLM on Ampere; a heterogeneous pairing of H20 for attention and L40S for experts gives 3.24× in throughput per dollar; the company claims a 1.5–2× reduction in serving cost, and the code is not public. Step-3 at StepFun (July 2025): MFA attention is deliberately tuned to an intensity of 128 to land in the profile of cheap accelerators, and the exchange goes through StepMesh — RDMA without a single streaming multiprocessor; measured 3,910 tokens per second per GPU at 50 ms TPOT and 4K context on 32 Hopper GPUs against 2,324 in DeepSeek-V3's profile, while the cost table — $0.055 per million tokens on H800 and $0.040 on H20 against $0.068 and $0.128 for V3 — is theoretical, at peak utilization. Huawei Cloud (August 2025, updated March 2026) serves attention, FFN, and experts as separate pools on a 384-chip Ascend 910C supernode: 2,400 tokens per second per chip at 50 ms — over a proprietary shared-memory fabric with no commodity equivalent.

What the 2026 simulations say

Researchers at Georgia Tech (May 2026) swept the design space on 128 B200s: under strict SLOs attention–expert disaggregation sustains about 4,000 tokens per second on DeepSeek-V3.2 where the co-located setup does not fit at all — at a ratio of 2 attention nodes to 126 expert nodes for a 524K-token prefix. But the same work shows that as long as chunked prefill fits in a co-located pool, it keeps the raw-throughput advantage. Imperial and Cambridge (August 2026) went further — four-stage specialization of prefill, decode, attention, and FFN gives up to 1.92× on agentic traces and 1.29× on chat, but only at input/output ratios of 10 or more; on H100 and A100 'plain P/D matches or beats it for six of eight models', splitting attention from FFN alone on GPUs never wins, and quadrupling the active experts cuts the benefit by up to 73%. ExpertPlex (July 2026) proposed the inverse cut — shared experts for prefill and decode with separate attention — and got 2.01× the goodput of instance-level disaggregation on H800. That is a direct argument against P/D being the end state for MoE.

Stage silicon: the chip died, the idea survived

On 9 September 2025 NVIDIA announced Rubin CPX — 'the first CUDA GPU purpose-built for massive-context AI': 30 PFLOPS in NVFP4, 128 GB of GDDR7 instead of HBM, '3× faster attention than GB300 NVL72', an NVL144 CPX rack at 8 exaflops and '$5 billion in token revenue for every $100 million invested', availability 'at the end of 2026'. SemiAnalysis sized the chip the next day at 2 TB/s over PCIe with no NVLink and about 800 W, with GDDR7 five times cheaper than HBM per gigabyte. It was silicon precisely for the prompt-processing stage of the previous article: lots of compute, little bandwidth. By GTC in March 2026 the chip had vanished from every document; 'Rubin CPX was dead within three months of being announced', Glenn Lockwood summarized, the product page returns 404, and there is no official statement.

Its place was taken by the opposite idea. After the licensing deal with Groq (24 December 2025; NVIDIA's fourth-quarter report shows a $13 billion outflow labeled 'Groq'), NVIDIA presented Groq 3 LPX: attention and prompt processing stay on Vera Rubin GPUs with HBM, while FFN and experts move to LPUs with 500 MB of SRAM and 150 TB/s per chip; 256 chips in a rack give 128 GB of SRAM. In its blog NVIDIA calls the scheme 'attention–FFN disaggregation' outright and promises '35× higher throughput per megawatt' versus GB200 NVL72 at 400 tokens per second per user — without a single independent measurement. AMD answered by acquiring Taalas (August 2026) for decode in Helios racks and by partnering with Cerebras: Helios for prompts, the wafer for decode, 'up to 5× tokens per second per watt' — modeled, not measured. AWS announced the same pairing of Trainium and CS-3 in March 2026. Silicon specialization survived the cancellation: the bandwidth-poor stage chip died, the capacity-poor layer chip lives.

The state of the open engines in August 2026

Phase disaggregation in vLLM is still 'experimental and subject to change', although the NIXL connector has closed heterogeneous TP, MLA, and hybrid models, with FP8 and NVFP4 KV still on the plan; two proposals for attention–FFN disaggregation were closed as stale. SGLang dropped the label from P/D, but its AFD branch is 'not ready for speed benchmarking'. Dynamo reached version 1.0 (16 March 2026) with a claimed 'up to 7×' requests on Blackwell per InferenceX and a user list from ByteDance to CoreWeave; llm-d shows about 3,100 tokens per second per B200 decode GPU under wide expert parallelism, and AWS measured +70% at 128 streams. ByteDance described autoscaling a fleet of tens of thousands of H20 and L20 GPUs with ratios from 1P:5D to 9P:1D, where decode speed is the only signal — 41.3% fewer GPUs on one multimodal service. That is already the norm; the frontier is the deeper cut.

The weightiest evidence for this direction is not a paper but a price list. Every major API prices prompt caching as a storage tier: a read costs a tenth of input at Anthropic, OpenAI, Google, and Alibaba and a thirtieth at DeepSeek; writes cost more than input; Google bills per hour of storage. Lifetimes map onto physical tiers. Minutes are DRAM: five minutes at Anthropic and Alibaba, five to ten in memory at OpenAI. Hours are local SSD: an hour at Anthropic, a day at OpenAI 'by offloading the key/value tensors to GPU-local storage', as Microsoft's documentation literally puts it. Days are distributed storage: at DeepSeek the on-disk cache 'is cleared usually within a few hours to a few days'.030303In August I covered the PagedAttention paper itself in the channel: back then the KV cache was fixed the way process memory is fixed — pages, a table, copy-on-write. Three years on, the same object is priced as storage with a tariff for reads, writes, and hours; the next step is a contract about who owns a block.Knizhny kub · vLLM and PagedAttention03In August I covered the PagedAttention paper itself in the channel: back then the KV cache was fixed the way process memory is fixed — pages, a table, copy-on-write. Three years on, the same object is priced as storage with a tariff for reads, writes, and hours; the next step is a contract about who owns a block.Knizhny kub · vLLM and PagedAttention

Under the tariffs sit systems that left the lab in 2026. Mooncake Store (March 2025), with a transfer engine at 87 GB/s over four 200G ports and 190 GB/s over eight 400G ports, is listed in the PyTorch ecosystem with adopters from Moonshot and Alibaba Cloud to Ant, JD, and Tencent. Its joint benchmark with vLLM (May 2026) — 610 Codex traces with a median of 33 turns on Kimi-2.5 in NVFP4 and 12 GB200s: hit rate from 1.7% to 92.2%, 3.8× throughput, P50 TTFT 46× lower. The baseline is 'no pool at all', so this is an upper bound. The SSD tier (July 2026) was measured on a two-hour trace from a Qwen cluster: without it hits fell from 83% to 36% after DRAM ran out and TTFT rose from 6 to 16 seconds; with it, over 84% and at most 9.4 seconds. The same team's budget calculator gives 84% hits at 512 GB and 88.5% at 2 TB on a Claude Code trace, which is where the knee sits. The ideal prefill speedup at hit rate r is simple:

Going from 90 to 95 percent doubles the gain — which is why the fight is over the last few points, not the first tens. LMCache (version 0.5.4 of 20 August 2026) gained a fleet-scale coordinator and runs at Cohere via CoreWeave; SGLang HiCache promises, at Novita, −56% TTFT and a hit rate from 40 to 80%, and at Ant −84% TTFT versus recompute — by the partners' accounts; vLLM shipped a DRAM offloading connector (January 2026: TTFT 2–22× better on an H100 with 500 GB of memory, multi-tier in developer preview). NVIDIA builds the KV block manager in Dynamo with four tiers from HBM to cloud object storage, but its documentation has no benchmarks, and the partner figures — '27× faster loading' at DDN, '4–10×' at WEKA — arrive without baselines. DualPath (February 2026, with DeepSeek participation) showed that KV can live in the 3FS distributed file system with no DRAM at all: the prefill node's 400G NIC saturates, so blocks are routed through decode nodes — 1.96× within SLO on 48 prefill and 96 decode nodes.

Reuse beyond the prefix and learned compression

Prefix caching reuses only the beginning. CacheBlend (EuroSys 2025) stitches chunk KV at any position, recomputing the 10–15% of tokens with the largest deviation: TTFT 2.2–3.3× lower at quality within 0.02 F1 — on question answering and summarization, not on long reasoning; since August 2026 LMCache looks such chunks up across the whole fleet. EPIC (ICML 2025) recomputes at most the 32 leading tokens of each chunk and loses 0–7% accuracy — not zero for production. The boldest bet is Cartridges from Stanford (June 2025): train a small corpus KV cache on synthetic dialogues and load it instead of the text; on Llama-8B that is 38.6× less memory and 26.4× higher throughput than the ordinary context. Amazon reproduced the idea in June 2026 on Qwen3-8B and collections over a million tokens — within 2–6 points of the full context at 10× compression — and showed that naive composition of several cartridges collapses to chance. The authors themselves write that self-study is 'much more costly than simply running standard ICL pre-fill' and pays off only where a corpus is queried many times.

Compressing a cache that already exists split into what works and what does not. KIVI-style 2-bit KV quantization loses almost nothing; TurboQuant loses 35% on summarization; token eviction — SnapKV, H2O, PyramidKV — loses 8–15% on summarization, fails in multi-turn context and on long reasoning, and the Bayreuth benchmark (2026) sums up: 'compression ratio alone is a poor predictor of end-to-end performance'. Production went the architectural way: MLA, cross-layer sharing of K and V in Gemma 3n, hybrid attention in DeepSeek-V4 and Kimi K3; the TransMLA and MHA2MLA converters turn GQA models into MLA with 92–93% compression after fine-tuning — on 7B-class models. The asset shrinks faster than the storage tiers under it mature.

A shared cache is a security boundary

The previous article called a cache hit a timing side channel. In a year it became a measured fact: a Stanford audit (ICML 2025) found a global, cross-user cache at seven providers including OpenAI; attacks such as PromptPeek recover other people's prefixes 95–99% of the time; SafeKV defends in over 94% of cases while cutting the isolation overhead from 50 to 12% of TTFT. Non-prefix reuse opened a new channel: fusion in vLLM with LMCache gives a 104-millisecond gap, and in bounded entropy extraction reaches 100%. 'Shadow in the Cache' (NDSS 2026) went further — inverting KV tensors reconstructs the input with BERTScore 1.0 at the first layer. Once KV sits on a shared SSD or in a CXL pool, it becomes a new class of data at rest.

The rule from the previous article — a format without a fast kernel only saves memory — has not gone anywhere. What is new is that four bits now have tensor cores and models that are born in those four bits. There are two formats. NVFP4: E2M1 elements, a block of 16, an E4M3 scale per block and an FP32 scale per tensor — about 4.5 bits per parameter including the scales. MXFP4 from the OCP specification: a block of 32 and an E8M0 scale, that is, a power of two — 4.25 bits. Blackwell computes both: an independent measurement on B200 (December 2025) put FP4 at exactly 2.01× FP8 at the instruction level; GB300 NVL72 has 1,080 dense petaflops in FP4; Vera Rubin NVL72 has 3,600 petaflops NVFP4 at 50 per GPU and 'one-tenth the cost per million tokens' — a vendor claim.

The models followed the hardware. gpt-oss (August 2025) shipped with MoE weights in MXFP4 — 4.25 bits per parameter for over 90% of the parameters, so that 120B fits in one 80 GB card, and every evaluation on the card was run in that format. Kimi K2 Thinking (November 2025) applied quantization-aware training in post-training and got INT4 for experts 'losslessly' with a '2× speed-up in low-latency mode'; the motivation was stated honestly — thinking models 'suffer performance degradation with quantization' because of 'excessive decoding lengths' — but every benchmark is reported in INT4 and there is no BF16 ablation; K2.5 and K2.6 use the same method. DeepSeek-V3.1 (August 2025) moved to the UE8M0 scale 'for the next generation of domestically produced chips', and DeepSeek-V4 (April 2026) stores routed experts in FP4 after post-training QAT, computes the attention indexer in FP4, and is validated on NVIDIA and Huawei Ascend — with the caveat that 'peak FLOPs for FP4×FP8 are currently the same as FP8×FP8' and 'could be 1/3 more efficient on future hardware'. NVIDIA requantized V4-Pro into NVFP4 and got GPQA Diamond 89.33 against 89.49 for FP8. Nemotron 3 Ultra (June 2026) was pretrained entirely in NVFP4 on 20 trillion tokens: the loss gap when switching to BF16 at the 5T, 10T, and 16T marks is under 0.4%, by NVIDIA's own ablation; in the 12B paper a year earlier MMLU-Pro matched (62.58 versus 62.62) but code slipped (56.67 versus 59.52), and MXFP4 on an 8B model needed 36% more tokens.040404Back in February 2025 I covered YC's short breakdown of DeepSeek's engineering in the channel: FP8 training with a fix for error accumulation was already the headline. Against that background, FP4 in V4 looks less like a leap than the next rung of the same ladder — and it needs its own kernels just the same.Knizhny kub · DeepSeek's engineering unlocks04Back in February 2025 I covered YC's short breakdown of DeepSeek's engineering in the channel: FP8 training with a fix for error accumulation was already the headline. Against that background, FP4 in V4 looks less like a leap than the next rung of the same ladder — and it needs its own kernels just the same.Knizhny kub · DeepSeek's engineering unlocks

Where FP4 pays and where it only shrinks

Kernels decide. An independent measurement on Blackwell (January 2026) gives FP4 for MoE 3.54× over BF16 at batch 4096, 2.23× at batch 128, and 0.86× — that is, slower than BF16 — at batch 1 in vLLM because of activation quantization. The ISTA and Red Hat group concludes that 'FP4 doesn't automatically outperform INT4', while its kernels give 2.2× end-to-end on B200 versus FP16. The gain is concentrated in expert matrices at high occupancy — prompt processing and large decode batches; at batch 1 only the weight-read savings remain, and only if the kernel leaves activations alone. An FP4 KV cache exists only in TensorRT-LLM on Blackwell with FP8 weights, is marked experimental in SGLang, and vLLM documents FP8 only.

Reasoning quality is a separate check. The COLM 2025 study (April 2025) found W8A8 and W4A16 lossless, while aggressive four bits for activations and KV cost 2.33% at 32B and over 10% at 1.5–7B. Meta in May 2026 described the failure mode: 3-bit AWQ on R1-Distill-Qwen-1.5B drops MATH-500 from 85.6 to 47.0 while the chain grows from 5.2 to 23.4 thousand tokens — up to 52% of failed answers reach the correct solution and never commit to it. ReQAT (ICML 2026) showed that training cures it: on R1-Distill-Qwen-14B AIME falls from 56.83 to 50.13 after plain NVFP4 for weights, activations, and KV and rises to 65.63 after QAT aligned to reasoning traces. Two bits without the cure — 17.2% on MATH-500 for Qwen3-8B. MLPerf holds the bar at 99% of FP16 accuracy, and in September 2025 NVIDIA passed it in NVFP4 across all DeepSeek-R1 and Llama submissions, while in April 2026 GB300 NVL72 showed 9,821 tokens per second per GPU.

Below four bits there is no evidence at scale. BitNet b1.58 2B4T (April 2025) is 2.4 billion ternary weights, with bitnet.cpp kernels 2.37–6.17× faster on x86 at 72–82% energy savings; no official ternary model above 2.4B exists as of August 2026, and neither does ternary silicon. Meta's ParetoQ (NeurIPS 2025) showed that ternary, 2-bit, and 3-bit weights beat 4-bit on accuracy per unit of size at up to 8B — on commonsense tasks, not long chains.

Decode buys bytes per second. Of everything shipping in 2026, only two things change the bytes-per-token economics of mainstream models. The first is HBM4 on Rubin: from 8 to a target of 22 TB/s per GPU at the same 288 GB, with SemiAnalysis's caveat that suppliers will deliver about 20 at first; the racks are going into five clouds. The second is silicon that trades capacity for bandwidth by keeping weights in SRAM: the Cerebras WSE-3 wafer with 44 GB and 21 PB/s, the Groq LPU with 500 MB and 150 TB/s per chip, the d-Matrix Corsair card with 4 GB and 300 TB/s. The interconnect becomes part of the device: 256 LPUs in a rack give only 128 GB of SRAM, and a 405B model in FP16 needs on the order of nineteen Cerebras wafers.050505I covered the history of the TPU in the channel: in 2013 Google calculated that without its own silicon it would have to double its number of data centers, and built a chip for one operation — matrix multiplication. Today's bets on SRAM and compute-in-memory follow the same logic, except the scarce resource is no longer operations but bytes per second.Knizhny kub · the history of the TPU05I covered the history of the TPU in the channel: in 2013 Google calculated that without its own silicon it would have to double its number of data centers, and built a chip for one operation — matrix multiplication. Today's bets on SRAM and compute-in-memory follow the same logic, except the scarce resource is no longer operations but bytes per second.Knizhny kub · the history of the TPU

Independent measurements exist only for those selling tokens through an API. Artificial Analysis in August 2026 shows on gpt-oss-120b: Cerebras at 1,733 tokens per second for $0.15 per million, SambaNova 699, Groq 474 for $0.06 — against low hundreds at GPU providers, of which CoreWeave is cheapest at $0.04; on Llama 3.3 70B — Groq 307, SambaNova 291, Google Vertex 131. In MLPerf v5.1 and v6.0 neither Cerebras, nor Groq, nor SambaNova, nor d-Matrix, nor Tenstorrent submits. A separate evaluation of GroqRack versus GPUs (2026) matches intuition: GPUs win prompt processing and batched decode, Groq wins single-stream decode, without batching.

Cerebras went from demos to contracts in a year: a multi-year 750 MW deal with OpenAI (January 2026), GPT-5.6 Sol in 'Ultrafast' mode at up to 750 tokens per second in a limited preview (August), first CS-4 shipments in the third quarter of 2026 with a promise of '30× faster than GPU systems' and no named baseline, an IPO in May. SambaNova claims 895 tokens per second on Llama 3.3 70B for SN50 against 184 on B200 in its own test and raised a round at an $11 billion valuation. Tenstorrent sells Galaxy Blackhole for $110 thousand: 6.2 GB of SRAM at 2.9 PB/s and a terabyte of GDDR6 at 16 TB/s — a bet on cheap capacity, not on per-chip bandwidth. Etched promises Sohu racks 'this summer' with no numbers; Taalas hardwires an 8B model's weights into silicon, gets 16,960 tokens per second — and becomes part of AMD. The hyperscalers' own chips — Ironwood at 7.37 TB/s, Trainium3 at 4.9, Maia 200 at 7 — sit at the same point on the chart as HBM GPUs: a bet on price and ecosystem, not on bytes per token.

Compute in memory: prototypes and simulations

Processing-in-memory promises the most radical shift — multiply weights by a vector where they lie. As of August 2026 no DRAM module with compute ships in volume: SK hynix's AiMX at CES 2026 is still a prototype, Samsung's LPDDR5X-PIM was measured by emulation on a smartphone, and decode on a real device 'could not be measured'. The only result measured on a prototype is NELSSA (SK hynix and KAIST, MICRO 2026): four CXL devices with processor cores beside an H100 give up to 5.5× tokens per second at 128K–1M context versus one GPU on Llama3-8B. Everything else is cycle-accurate simulation with multipliers of 3 to 15 against an H100, not Blackwell, and mostly at long context, where the bytes are spent by attention rather than FFN.

CXL memory pools and optics solve other problems. Samsung in June 2026 measured a CXL 2.0 pool on eight RTX PRO 6000 GPUs: 92.3% of local DRAM performance at 35–47 GB/s — a capacity tier for KV, two to three orders of magnitude below HBM. Marvell's CXL 3.0 switch at 4 TB/s aggregate samples from the third quarter. Co-packaged optics are real — Quantum-X Photonics is available, Spectrum-X Photonics comes in the second half of 2026 with '5× power efficiency versus pluggables', Ayar Labs raised $500 million and joined NVLink Fusion, Marvell closed its purchase of Celestial AI with revenue only from 2028 — but optics do not change bytes per token; they change how many HBM pools can be tied into one domain per watt.

Test-time compute inverted the shape of a request. In Alibaba's production traces (NSDI 2026) the reasoning segment is on average four times longer than the answer, with a bimodal distribution. DeepSeek disclosed its system in February 2025: 608 billion input tokens per day at a 56.3% cache hit rate and 168 billion output tokens; a four-node prefill unit against an eighteen-node decode unit; $87 thousand of daily cost against $562 thousand of theoretical revenue. o3 on ARC-AGI-1 (December 2024) scored 75.7% at $26 per task and roughly 55 thousand tokens per attempt — and 87.5% at $4,560 per task with 1,024 attempts. By June 2026 Artificial Analysis counts the cost and time of an index task: Opus 4.8 at maximum effort — $1.78 and 6.4 minutes, GPT-5.5 at xhigh — $0.99 and 3.7 minutes, DeepSeek V4 Pro — $0.04; time per task at the frontier runs from one and a half to thirteen and a half minutes. The unit of account has moved from the token to the task.060606In Karpathy's 2025 review, which I covered in the channel, the first item is RL on verifiable rewards and the new lever of 'let the model think longer'. For serving, that lever means the length of the answer is now set by a policy, not by the user, and the scheduler is the last to find out.Knizhny kub · Karpathy's 2025 year in review06In Karpathy's 2025 review, which I covered in the channel, the first item is RL on verifiable rewards and the new lever of 'let the model think longer'. For serving, that lever means the length of the answer is now set by a policy, not by the user, and the scheduler is the last to find out.Knizhny kub · Karpathy's 2025 year in review

Volumes grow faster than prices fall — and alongside them. At I/O on 19 May 2026 Google cited over 3.2 quadrillion tokens per month, seven times more than a year earlier; OpenRouter showed in January 2026 that the reasoning-model share of tokens grew from near zero in early 2025 to over half, and the average prompt from 1.5 to over 6 thousand tokens. Epoch AI (March 2025) records the price of a fixed capability level falling 9–900× per year; 'The Price of Progress' (November 2025) puts it at 5–10× per year per unit of capability, yet running the frontier itself gets 3–18× more expensive per year due to scale and reasoning; the 'Tiered Super-Moore's Law' analysis (March 2026) estimates the reasoning premium at 31.5× the price of ordinary models.

Budgets are soft, parallel attempts are opaque

Budget control in APIs has matured to step six, but every knob is soft. Anthropic declared budget_tokens 'a target rather than a strict cap', then replaced it with effort levels from low to max and added a task budget (beta since March 2026) — 'a soft hint, not a hard cap'; the only hard limit is max_tokens, and changing the effort invalidates the cache. Gemini moved to discrete thinking levels, OpenAI to reasoning_effort from 'none' to 'max', and GPT-5.6's Pro mode 'aggregates the work it performs into a single answer' with the warning to 'expect higher token usage'. An important detail for memory: prior reasoning now stays in context and is billed as input — at Anthropic from version 4.6, at OpenAI by default for 'all turns'. Parallel attempts reached products at OpenAI (GPT-5 Pro 'makes use of parallel test time compute' — algorithm undisclosed) and at Google (Deep Think with 'parallel thinking' and a daily prompt quota); Anthropic described a variant with 256 samples and a learned scorer that reached 84.8% on GPQA, and did not deploy it.

Research promises multiplicative gains — on open models up to 14B. Certaindex (2025) measures answer stability and exits early: up to 50% compute saved and 3.3× throughput with no accuracy loss. SpecReason verifies reasoning steps with a small model — 1.4–3.0×. Length-penalized training (ALP, JET) trims 46–50% of tokens on 1.5B models. Evicting thinking tokens from KV — R-KV, ThinKV, ForesightKV — keeps 5–10% of the cache at preserved accuracy and gives 5.8–6.6× throughput; no such eviction is visible in production, and vendors are moving the other way. Latent reasoning without tokens — Geiping et al.'s recurrent depth (3.5B, February 2025), Meta's Coconut, Ouro at 2.6B, LOTUS at 3B (June 2026), WeLM-HD4 at 617B (July 2026, four streams per token, no deployment) — has not reached a single frontier product. Letta's sleep-time compute (April 2025) — five times less compute at equal accuracy — lives in Letta's product; the major APIs do not have it.

The first public agent traces appeared in 2026, and they agree on one thing. TraceLab from the University of Washington (June 2026) collected 4,265 Claude Code and Codex sessions from 43 developers: one human request carries 0.9 of its own steps and 7.8 tool-initiated steps with 10.8 tool calls; the median prefix is 126K tokens for Claude and 116K for Codex, about 860 are appended, and the output is 252 and 184. Microsoft (July 2026) published a characterization of GitHub Copilot over one week in June: 3.2 million users, 13.5 million sessions, 761 million model calls; 87% of calls are agent-initiated. Inferact, together with vLLM (May 2026), released 610 Codex traces on SWE-bench Pro with a median of 33 turns. Agent traffic is prompt processing with enormous repetition: input to output from 131:1 to 630:1.

The prefix already works — as long as the next call quickly lands on the same node. Hits of 95.7% at TraceLab and about 98% within a turn at GitHub Copilot break at human pauses: 55% at turn boundaries, misses after idle over five minutes and nearly all after an hour, −66 percentage points after context compaction, and 8% after a model switch. At TraceLab the model takes 41% of request time and tools 60%, yet the median tool call lasts 0.3 seconds, and the 4.9% of calls longer than a minute account for 92% of all tool time. Concurrency within a session at GitHub Copilot has a median of 1.15, with 93% of tool batches single: the sub-agent burst is not yet visible in real traces, and Anthropic's '15× the tokens of chat' for multi-agent systems remains a claim. Manus put it a year ago: 'KV-cache hit rate is the single most important metric', at 100:1 and a tenfold price gap between cached and fresh tokens.070707I covered Boris Cherny's account in the channel of how the team cut more than 80% of Claude Code's system prompt for a new model generation. Serving sees a second side to that: the system prompt is also the stable cache prefix, and every edit to it changes not just behavior but the hit rate.Knizhny kub · Boris Cherny on the Claude Code prompt07I covered Boris Cherny's account in the channel of how the team cut more than 80% of Claude Code's system prompt for a new model generation. Serving sees a second side to that: the system prompt is also the stable cache prefix, and every edit to it changes not just behavior but the hit rate.Knizhny kub · Boris Cherny on the Claude Code prompt

Schedule the session, not the request

The research line runs from Parrot (OSDI 2024), where the application annotates semantic variables and the service sees the call graph, through Autellix (February 2025) with the program as the scheduling unit — 4–15× throughput at equal latency against vLLM 0.6.1 with an FCFS queue, mostly from removed head-of-line blocking — to Continuum (November 2025, ICLR 2026): on a tool call the KV is pinned in HBM for a time equal to the tool's expected duration from its empirical distribution. On 500 SWE-bench Verified tasks that is 8.18× lower mean job completion time than SGLang 0.5.5 and Dynamo 0.7; a SWE-bench tool call lasts 925 ± 3,550 milliseconds over 10.9 turns. KVFlow (NeurIPS 2025) evicts by the agents' step graph — 1.83–2.19× over SGLang; CacheScout (July 2026) learns agent-to-agent transitions online and adds 10–18 percentage points of hits over Continuum; ConServe (June 2026) makes the whole conversation the placement unit — −51% P95; PASTE (March 2026) runs tools speculatively during decode — −43.5% task time. SAGA (May 2026) is the only work at 64 GPUs: task completion time on SWE-bench and WebArena is 1.64× lower, but peak throughput is 30% below throughput-optimal scheduling. That is the most honest published price of atomic scheduling.

The server side is moving from cache to session. Anthropic holds 5 minutes and an hour; OpenAI — a day by default for gpt-5.5 at no premium, while GPT-5.6 introduces explicit breakpoints, an exact 30-minute TTL, and paid writes. Claude Managed Agents (beta since April 2026) move the session, the sandbox container, and the history to the server at $0.08 per active session hour plus tokens — with no published KV lifetime. Prism (OSDI 2026) shows what happens to a fleet of many models that idle over 70% of the time: ballooned GPU virtual memory redistributes KV among them, and the authors report deployment on over 10 thousand GPUs with 3.89× tokens per GPU at an unnamed company.

Multimodal inference is not text inference with a preprocessing step but a different resource profile at every stage. Microsoft's ModServe (SoCC 2025) measured on production traces: image encoding takes from 25% of TTFT on InternVL-26B to 79% on Llama 3.2 with cross-attention, and one 896×896 image costs from 1,280 to 7,290 tokens depending on the architecture. Separate encoder and text pools with modality-aware scaling gave 3.3× and 5.5× throughput over monolithic vLLM on 128 A100s at 25–41% cost savings. Huawei (ICML 2025) pushed E/P/D to a 71% TTFT reduction and a 22× larger batch, and its EPD-Serve on Ascend (January 2026) to 57–69% over plain P/D. HeteroServe's argument (March 2026) explains why the cut runs along the modality boundary: between the encoder and the model travel megabytes of embeddings, not gigabytes of KV.

Video turns context into a time budget. By the Gemini documentation a frame costs 258 tokens at one frame per second plus 32 audio tokens — about 300 tokens per second, so a one-million-token context is roughly one hour of video. My arithmetic at a KV footprint typical of 8B-class models, 128 KiB per token: an hour of video is on the order of 130 GiB of cache, more than one GPU. Qwen3-VL (November 2025) answers with 32× spatial and 2× temporal compression at a native 256K context; the Qwen3-Omni model card shows the price honestly: 78.85 GB of memory for 15 seconds of video and 144.81 GB for 120 seconds — about 0.63 GB per second. In the engines the encoder stage is only emerging: vLLM's E/P/D tracker has been open since 15 August 2026, with encoder-cache connectors over Mooncake, NIXL, and LMCache merging one by one; SGLang has no encoder stage and removed its dead code on 21 August; TensorRT-LLM supports 'only decoder-only models', and its E/P/D tests are unstable. The separate vLLM-Omni project serves omni models, diffusion, and robot policies with 'fully disaggregated' stages and an experimental duplex mode.080808In the ETH lecture I covered in the channel, LeCun insists that a world model should predict state, not pixels. For inference that is not philosophy: generating a frame means paying for every pixel every 40 milliseconds, predicting state means paying for an abstraction. Which of the two fits into the frame clock for a hundred users, nobody has shown yet.Knizhny kub · LeCun on JEPA and world models08In the ETH lecture I covered in the channel, LeCun insists that a world model should predict state, not pixels. For inference that is not philosophy: generating a frame means paying for every pixel every 40 milliseconds, predicting state means paying for an abstraction. Which of the two fits into the frame clock for a hundred users, nobody has shown yet.Knizhny kub · LeCun on JEPA and world models

Visual-token compression: not every 'fewer' is faster

The line of FastV (ECCV 2024, −45% FLOPs), SparseVLM (ICML 2025, −54% FLOPs at 97% accuracy), PyramidDrop (CVPR 2025), and VisionZip (8× faster prefill) looked like the flagship until DART (2025) showed that attention-importance selection 'usually results in inferior performance than random', while duplication-based pruning removes 89% of tokens at 1.99× end-to-end speedup. A 30-example pilot from August 2026 added the main point: 'fewer visual tokens do not guarantee lower end-to-end latency' — autoregressive probes stayed slower than the full baseline, and only pre-vision routing won, because only it skips preprocessing and encoding entirely, even with eight times less token reduction. GSTEP prunes 75% of video tokens for 1.17×. None of these methods is documented as a feature of vLLM, SGLang, or TensorRT-LLM, and modern dynamic-resolution models already allocate tokens themselves — the inference-time headroom is structurally smaller than on LLaVA.

Duplex speech and frame generation: the SLO is a stream

For real-time speech the SLO is not a token but a continuous stream. Kyutai's Moshi keeps two parallel streams at 160 milliseconds of latency in theory and 200 in practice; Qwen3-Omni (September 2025) replaced block diffusion in its Thinker–Talker architecture with a light causal convolution so that synthesis streams from the first codec frame — 234 milliseconds to the first packet in theory, and vLLM serves only the Thinker. At OpenAI the transport gives about 100 milliseconds over WebRTC and 200 over WebSocket with a 60-minute session cap; Gemini Live is still in preview, and its audio tariff is the only thing from which the price of an hour of duplex speech can be derived: about $1.38 per user by my arithmetic from $3 and $12 per million. LiveServe (June 2026) showed that the scheduler must see playback progress and barge-in: on vLLM-Omni P90 first-packet latency is 1.55× lower and completed requests 1.15× higher.

Frame generation is the third regime. World models such as Genie 3 (5 August 2025: 24 frames per second at 720p, a few minutes of interaction, about a minute of memory) became by summer 2026 a paid product — Project Genie in the Google AI Ultra subscription at €99.99 or €219.99 a month in 140 countries — with no disclosed cost per hour, session quota, or number of concurrent users. World Labs runs RTFM 'on a single H100' and states the scale of the problem: an interactive 4K stream at 60 frames per second is over 100 thousand tokens per second. The open reproductions — Matrix-Game 3.0 at 40 frames at 720p on a 5B model and a four-player multiplayer at 20 frames on one B200 — are single-user. The gap between batch and interactive generation is measurable: a four-second Sora 2 clip on Azure took 67 seconds to produce, 17 times slower than real time, and Veo 3 costs $0.40 per second; diffusion-step caching such as TeaCache gives 4.41× against an undistilled baseline and almost nothing on four-step models, where Chorus extracts +45% from inter-request reuse. Those two orders of magnitude are being closed by distillation, not by caching.

Power has become the limit on fleet growth — but not on operating what already exists. By the IEA's update (April 2026) data centers consumed 485 TWh in 2025, 17% more, and AI-focused ones 50% more; by 2030 — around 950 TWh, about 3% of global demand, and 15–27 GW of gas generation right on site, mostly in the United States; transmission lines take four to eight years, and transformer lead times have doubled. Epoch AI (January 2026) estimates AI data-center capacity at the end of 2025 at about 30 GW — comparable to New York State's peak — and the path to a gigawatt site at one to three and a half years. SemiAnalysis (June 2026) reckons US data centers will add 21 GW in 2026 and 84 in 2030 against roughly 15 GW of net new grid capacity per year, with turbines and transformers waiting three to four years instead of the customary eighteen months; hence the forecast of '40+ GW behind the meter by 2028'. Andy Jassy in February 2026: AWS 'could actually grow faster if supply were available'; Microsoft's CFO: 'short now for many quarters... power and data center space'. The big four's 2026 capital spending runs from $120 to $200 billion each.090909The most radical reaction to the power limit I have covered in the channel is Starcloud with data centers in orbit. Their founding hypothesis matches this section: the constraint is no longer chip output but available energy. For now it is a bet on the future economics of launches, not a proven substitute for ground sites.Knizhny kub · Starcloud and data centers in orbit09The most radical reaction to the power limit I have covered in the channel is Starcloud with data centers in orbit. Their founding hypothesis matches this section: the constraint is no longer chip output but available energy. For now it is a bet on the future economics of launches, not a proven substitute for ground sites.Knizhny kub · Starcloud and data centers in orbit

What can a scheduler do? Decode is memory-bound, so the lever is frequency, not a power cap. Work from Erlangen (May 2026) showed on an H200 that decode draws 137–300 W of 700 and a power cap 'never triggers', while locking the streaming multiprocessor clock recovers up to 32% of decode energy at minimal throughput loss, and MLA and linear attention halve request energy versus GQA. Controllers show savings proportional to slack: GreenLLM (August 2025) — up to 34% over default DVFS with under 3.5% extra SLO violations, but 27.5% at one request per second and only 6.8% at ten; DynamoLLM (HPCA 2025) — 53% energy under an SLO of 'P99 within five times the unloaded latency', mostly from diurnal under-utilization and simulated at fleet scale. It is more honest to frame power as a schedulable resource: AMD's RAPID (January 2026) shifts it between prefill and decode nodes in fractions of a second and gets 1.7× QPS per watt at a node budget 20% lower; KAIROS (April 2026) uses the context of the agent loop as a signal and cuts power by 27–46%; AFlex (August 2026) gives attention and FFN different frequencies — −49% energy per token.

Learned control reached the releases but not the benchmarks

Lodestar (ByteDance and UIUC, May 2026) replaces the router's hand-tuned heuristics with an online-learned TTFT predictor: 1.41× lower mean TTFT against a cache-and-load heuristic and up to 4.4× on a heterogeneous fleet of old A30s and V100s; a 16-GPU testbed. TokenScale (December 2025) makes 'token velocity' across prefill, network, and decode the single autoscaling signal — SLO attainment from 50–88% to 80–96% at 4–14% savings. The Vidur simulator remains the working tool for capacity planning: under 9% error, a 70B configuration found in an hour on a CPU instead of 42 thousand GPU hours. In the open runtimes SLO-aware routing has reached releases: the Gateway API Inference Extension for Kubernetes in version 1.5 (April 2026) got deadline ordering and shedding by predicted latency, llm-d 0.9 (August 2026) an XGBoost latency predictor and an SLO-driven autoscaling guide; Dynamo's SLA planner requires profiling in advance, and SGLang has no learned policies. Not one official page publishes a benchmark for the predicted-latency path. The only production evidence of an 'intelligent runtime' is Prism, on the memory side.

Semantic response caching took a similar road: vCache (ICLR 2026) replaced the global similarity threshold with per-prompt thresholds under an error-rate guarantee — up to 12.5× hits and 26× fewer errors against static thresholds; production estimates are 40–60% hits in repetitive categories against 5–15% in volatile ones; LaCache (August 2026) formalized cache poisoning through collisions as a new attack.

Trust: confidential inference steps out of the first party

Confidential inference received three independent overhead measurements. ETH (September 2025): an H100 in confidential mode loses 4–8% of throughput, and the gap shrinks with batch size. Mozilla (May 2026): at a fixed request rate TTFT rises 22–28% and throughput falls 18–21% — 'reserve 15–25% additional capacity'. Phala: under 7%, near zero for large models, because the overhead is PCIe transfer. Nobody has numbers for Blackwell, and NVIDIA promises 'performance comparable to unencrypted models'. The directional signal came from Apple on 8 June 2026: Private Cloud Compute runs outside Apple silicon for the first time — on Google Cloud with NVIDIA confidential GPUs, Intel TDX, and the Titan root of trust under the same guarantees: stateless computation, no privileged runtime access, non-targetability, verifiable transparency. Google's Private AI Compute (November 2025) and Meta's Private Processing (April 2025) are the same model for their own products; Anthropic and Pattern Labs call their work 'a sketch of our research'; the clouds have confidential H100s but no Blackwell. Verifiability without a trusted environment — TOPLOC at 258 bytes per 32 tokens and zkLLM proofs in 15 minutes for 13B — remains research, and a survey from Imperial (May 2026) concludes that an end-to-end substrate for agentic systems 'does not yet exist'.

Let us put the ten directions on one map: horizontally — the ladder step from the first section, vertically — the size of the promise. The positions are my reading of the sources as of August 2026, not a measurement; the further right a point, the lower the risk, the higher it sits, the bigger the promise and the stricter the baseline it needs.

What follows from the map is not a shopping list but an order of experiments. First the trace, because every direction pays off only under a particular shape of load: hybrid attention above 128K context, attention–FFN disaggregation with MoE and fast interconnect, four-stage specialization at input/output ratios of 10 or more, speculation at low concurrency, an SSD cache at pauses of minutes and hours. Then the no-regret bets that win under almost any profile. Then the conditional ones, each with a measurable threshold. And a list of what not to do until the step changes.

An order of operations for a platform team

• •Capture a trace with the four distributions from the previous article and add to them the share of requests above 128K, the input/output ratio, prefix reuse as a function of pause length, and concurrency within a session.
• •For each direction — one experiment with an honest baseline: the same model, the same engine without experimental flags, SLOs on P95 and cost per task, not the best tokens-per-second figure.
• •No-regret bets: prefix caching with DRAM and SSD tiers sized from the trace; FP4 or INT4 for expert weights with task-accuracy validation; speculation while verification is memory-bound; clock locking on decode nodes.
• •Threshold bets: hybrid attention at a meaningful share of long contexts; attention–expert disaggregation only inside an NVLink domain; a separate encoder pool at a noticeable share of images; SRAM silicon as a service for latency-priced tasks.
• •What not to do: a diffusion model as the primary at high concurrency; evicting thinking tokens without validation on your own tasks; quantization below four bits for reasoning; designing around announced but unshipped silicon.
• •Revisit the map in six months on these signals: LPX and CS-4 shipments with independent measurements, the 'experimental' label coming off P/D in vLLM, MiniMax's next architecture report, frontier cost per task at Artificial Analysis, the first hit-rate SLA for day-long sessions.

• —The frontier is read through three questions: which resource, against which baseline, what must become true.
• —A multiplicative gain on the first step is a hypothesis; on the fourth it is someone's engineering fact; on the sixth it is your default expectation.
• —The most common mistake is multiplying factors taken from different steps and different concurrencies.

Trace → ladder step → experiment with a baseline → payback threshold → decision → review in six months

The map's main conclusion matches the previous article's, one step higher. Inference remains a distributed system in which the model sets the shape of state and the admissible numbers, silicon sets bytes per second and the interconnect domain, the runtime sets the order of work, and the workload — reasoning, agents, video — sets the shape of a request. The 2026 frontier does not cancel a single layer; it argues about the boundaries between them: where the cut runs, who owns the state, and what to pay for the next multiplier.