Homa: Why GPUs Wait for the Network (#AI)

I watched a recent talk by Stanford's John Ousterhout about Homa (John is a co-author of the Raft consensus protocol, as well as the author of the excellent book "A Philosophy of Software Design," which I have written about before). The talk's title advances the claim that this is “the end of TCP for AI clusters,” while the talk itself raises an interesting engineering question: how much time do expensive GPUs spend idle while a small message waits behind a large transfer? Ousterhout argues that short exchanges are becoming more important in inference and agentic systems: checking an entry in a distributed KV cache or coordinating the next computation step. Tail latency matters here: one late response can delay everyone participating in a synchronization barrier.

Distributed systems have long faced this problem (it literally continues the story of "The Tail at Scale," which I covered yesterday). When several servers send data to one receiver simultaneously, a queue builds up before its network port: incast. A short message has to wait too.

Homa proposes rebuilding transport around messages: — Know each message's length and favor those with fewer bytes left to transmit. — Control traffic from the receiver: the sender transmits an initial portion, then needs permission through grants. — Use switch priority queues to let short messages bypass large transfers.

In the benchmark shown, the author reports roughly 13 times lower p99 latency for short messages with Homa than with TCP. Large messages also benefit. A Linux module, tests, and measurement utilities already exist. But I would challenge the title and the breadth of the conclusions.

1️⃣ The presentation slide shows a network benchmark that demonstrates the effect of using the protocol It does not establish a 13-fold LLM speedup: we need application response times, tokens/s, and GPU utilization. The benefit depends on how much waiting actually comes from transport. 2️⃣ The industry has been working on this problem for years Google described production deployment of Swift, while SIRD explores how to coordinate receiver decisions when a shared link becomes the bottleneck. A comparison with TCP does not settle which transport is best. 3️⃣ The deployment boundaries are substantial: Homa targets networks within a datacenter. It requires application integration and network configuration; development of grpc_homa is suspended. There is plenty of integration work ahead.

Further reading: research papers and the technical rationale: — Homa, SIGCOMM 2018: protocol design. — Linux implementation, USENIX ATC 2021: measurements on 40 nodes; the page includes a PDF. — It’s Time to Replace TCP in the Datacenter: Ousterhout's case against TCP's architecture in datacenters.

At the end of the talk, the author invites people to experiment with Homa and offers help: ouster@cs.stanford.edu. A useful starting project would measure how much of the network benefit survives in a real AI workload. That is a result I would like to see.

#AI #Architecture #Engineering #Research #PlatformEngineering