Dylan Patel about AI-inference and co-design of software and hardware as a key to multiple growth (AI column)

Watched Sequoia Training Data release with Dylan Patel, founder of SemiAnalysis:Why Hardware-Software Co-Design Is AI's Real 100x" In fact, I follow the work of this company. (Like me. handler Previous Post “The Great GPU Shortage”). This interview with the founder of the company was taken just recently and famously highlights that improvements in the capabilities of genAI systems are based on the joint design of models and hardware: the model, kernels, runtime, network, memory, chip and data center cannot be optimized as independent parts. Separate growth in 2x in models, 2x in cores (kernels) And the "2x" in iron may not give 8- multiple growth, and the leap by an order of magnitude greater (100x)If the model and iron are designed together. Or, conversely, a good chip may look weak if the model and software are not suitable for it. (New models on old chips).

The most specific example in the release is InferenceX, live. SemiAnalysis benchmark for inference. According to Patel, point-in-time benches quickly become obsolete: models change almost every week, PyTorch, vLLM, SGLang, drivers and inference optimizations are constantly updated, and one dimension a month later already poorly describes the market. Therefore, every day InferenceX runs models on different types of iron and builds not one digit “who is faster”, but a curve throughput / interactivity.

This is an interesting look at the balance of bandwidth and token speed per user – batch tasks and interactive coding agents have different economies. If the user is waiting for an answer in the feedback loop, then the delay is expensive: you can pay more for fast mode, speculative decoding or a smaller batch size. If you need to run a pack of documents at night, it is more important tokens per second per unit of iron, not an instant reaction. In total, the “best model” and “best hardware” should be evaluated relative to workloads, delay budget, and cost of error.

Further Patel interestingly tells about the top labs and in his opinion OpenAI, Anthropic, Google and DeepSeek differ not only in the quality of models, but also in architectural form. Significant components differ

• Multiplication matrix dimensions
• Attention structure (attention)
• Expert structure at MoE (mixture of experts) Network topology and memory bandwidth
• and so on. In the end, all this determines where the model will work well. According to Patel DeepSeek, it is better to fall on Nvidia/Hopper/Blackwell-like world. (And now Huawei's Ascend.)TPUs are strong in other model classes, and Google is optimizing Gemini for its TPU generations. As a result, we see models becoming inseparable from the iron on which they will live.

This gives an interesting look at the benefits. (moat) Nvidia, which today doesn't boil down to everyone using CUDA. Models are already good at writing custom cores, and large labs have long been able to fork PyTorch or assemble their infrastructure. But it’s important that all open-source models, inference providers, RL companies, and downstream ecosystems are often optimized for Nvidia, simply because there’s already a lot of work models and bundles. If Google has a comparable open model ecosystem under the TPU, that could shift.

I think this is a good antidote to simple conversations about AI hardware. Every major player is looking for the optimal for themselves in the bundle of model architecture + infrastructure software + silicon + power. A dedicated chip can be beautiful in one area of the piplin and unsuccessful in a year if the architecture of the models has gone the other way. So standard computing hardware stays in the mainstream not because of market laziness, but because even big labs don’t always know which models will be running in a year’s time.

Patel gives interesting estimates and forecasts that look useful for understanding this market. The cost of inference per unit of quality drops by about 60x per year. The point is to compare not just the price of the token, but the cost of achieving a comparable level of quality. Intelligence per watt is not growing by 60x, but by about 40x per year. Inference will be one of the largest markets in the world, possibly more oil.. This is tied to the fact that the value of the work that AI can do will be a huge share of the economy. K 2030 OpenAI + Anthropic, according to his forecast, may have more 100 The GW computer will be shared, and then Meta, Google and others will be added to this celebration of life. K 2040 Inference deployments can be measured in terawatts, as follows from previous forecasts.

• In 2030 Data centers in space will be almost irrelevant: less 1% iron for inferens, but to 2040 They will potentially take up more than half of the computer’s increment.

In summary, the main conclusion of the interview is that AI infrastructure finally becomes system engineering. You cannot choose a model separately, buy iron separately, set up serving separately and then be surprised by the economy. You need to describe task classes, latency/cost curve, response quality, context size, network constraints, available power, capital constraints, and only then decide where fast mode is needed, where batch is, where the GPU is, where the TPU is, where its binding is.

In practice, this means that when building an AI product, it is worth measuring not the “cost of the token” in general, but the cost of useful work on a particular workflow. In one place, an expensive fast token pays off because it shortens the human feedback loop. Elsewhere, he's just burning the budget.

#AI #Engineering #Architecture #Infrastructure #Bigtech #Software #SystemDesign