Stanford MS&E435: Inference as a Production System

After the history of Google TPU and the conversation with Dylan Patel about hardware-software co-design, the second Stanford MS&E435 lecture adds the next piece of the puzzle. It is titled “The GPU Economy,” but the interesting question is not which chip will win. It is the point at which software becomes manufacturing: every new request starts the factory again.

The session is moderated by Apoorv Agrawal, the course instructor and an Altimeter partner. The guests are Altimeter founder Brad Gerstner and Sunny Madra, the former Groq president who joined NVIDIA with part of the team. Gerstner supplies the economic frame; Madra explains the hardware. Their perspective matters: this is an AI-infrastructure investor speaking with one of its suppliers, so the market numbers and forecasts deserve a healthy discount.

The technical core is inference as a heterogeneous workload. Prefill processes the input context, while decode emits tokens sequentially; different parts stress compute, memory, and latency in different ways. Instead of one “best GPU,” the result is a composed system:

— a GPU with HBM remains the flexible engine for heavy workloads; — the Groq LPU uses deterministic execution, compiler-managed SRAM, and predictable latency; — NVLink connects the accelerators while the runtime assigns each architecture the work that fits it.

According to Madra, their joint prototype produced 2.5 times more tokens within the same power envelope (although the lecture provides neither a full configuration nor a reproducible benchmark, so this remains a participant's claim). The principle itself is already visible in NVIDIA's product architecture: Rubin GPUs and Groq LPX are being designed as one inference system; NVIDIA provides more detail in its Groq 3 LPX announcement.

The product lesson matters just as much. Groq did not ask developers to fall in love with a new architecture first; it hid the hardware behind the GroqCloud API. The team then stopped fighting NVIDIA head-on, decomposed the workload, and found the layer where the two architectures complement each other. In deep tech, compatibility with the dominant platform can be more powerful than a clean benchmark victory.

The speakers also repeatedly describe NVIDIA as having “bought Groq for $20 billion.” The formal structure was more nuanced: the companies signed a non-exclusive licence, part of the team joined NVIDIA, and Groq remained independent with GroqCloud intact. NVIDIA's filing discloses $17 billion in total consideration and explicitly says that no equity, customer contracts, or existing products were acquired. In effect, this was the acquisition of part of the intellectual property plus the team.

Then comes the economics. The unit cost of inference falls, while reasoning systems and agents take more steps, call tools, verify their work, and run for hours. Cheaper tokens therefore do not necessarily resolve the GPU shortage; they may expand total demand. It is close to the Jevons paradox, except that matrix multiplications replace coal.

This is where I would push back on Gerstner: more tokens do not automatically mean more value. We have already discussed token burn as proof of work and non-zero marginal cost in AI pricing. A mature product metric is not tokens per task but cost per verified outcome: useful work including retries, review, and rework.

There is even an indirect connection to robotics. Physical AI needs more than tokens per watt: it needs successful actions per joule, p99 latency for the closed control loop, and demonstrated safety. In the Waymo review, the cost of failure was already measured in something other than tokens. That is where the inference factory ends not with a chat response but with a machine moving through the physical world.

I would watch minutes 16 through 35 closely: that is where the conversation stops sounding like a presentation of a grand AI future and becomes a strong discussion of systems engineering.

#AI #Engineering #Architecture #Infrastructure #Agents #Robotics