System Design Through the Lens of Reliability

The host notes that system design is treated as a sacred cow, though everyone can picture the design of an MVP-level app. Alexander answers with the effect that catches engineers at the moment of business success: the app was built fast, not reliably and not for scale, because premature optimisation is a sin and nobody likes to sin. Traffic arrives, the business asks for more features, and paralysis sets in. Andrey offers the metaphor of a deferred loan accruing interest; that, Alexander agrees, is how engineers explain to the business where new tasks came from. If the debt is not repaid in time, the team reaches technical bankruptcy, declaring that version 1.0 must be thrown out and 2.0 written instead. His example is Netscape, in the nineties as far as he recalls: while the codebase was rewritten over a year or two, competitors did not wait and the market was ceded.

Before the New Year Alexander took apart the counter-example: the Amazon manifesto on distributed computing announced at the latest AWS re:Invent, and what had happened to the company in the 2000s. The margins did not add up: too much went on infrastructure, on Sun servers, on running their own shop and logistics. Amazon froze business features for a year, if not longer, and spent it moving from Sun to Linux, then only five or six years old. Mid-move the dotcom crash hit and Sun servers could be had for pennies, but the commitment had been made; one top manager recalled they had money left for a couple of quarters. Operating costs fell by 80%, by 2001 the company knew how to live with Linux, caught the x86 virtualisation wave that VMware engineers made possible by substituting instructions in executable code, and turned what had been costs into AWS.

Legacy, Andrey reminds him, is not about unmaintainable code but about a shortage of knowledge of it: too much code, someone else's, a large team. The paradox is that a small system needs no reliability — it is a prototype for a demo — yet nobody wants to throw it away later, and the same codebase then absorbs both functionality and fault tolerance. Getting lucky is a poor business strategy, so the talk turns to principles. In the Google book Building Secure and Reliable Systems, he reckons, a third or more is given to trade-offs and non-functional requirements. The first is fighting complexity, design for understandability: if the system is a black box, throwing in resources and raising five instances stays a palliative — a flaky stateless component can be scaled until it falls over entirely. Hence his complaint about re-badged administrators in the SRE role; his company pushes them to learn the systems they support. The second is readiness for a shifting landscape: bare metal, virtualisation, containers, a workload orchestrator.

Complexity, Alexander says, is the main problem software builders fight, and he recommends a very thin book, A Philosophy of Software Design, by the creator of Raft. It grew out of Stanford courses where the author tested whether good design can be taught: he set an iterative project, reviewed it and showed where breaking the basic principles hurts flexibility; its value is the mix of high-level concepts and concrete pitfalls. From the same author comes the metaphor of deep and shallow modules: a module should hide complex logic behind a simple interface, whereas a Java getter and setter make a shallow module whose call is longer than the action itself. Asked where else to learn patterns, Alexander recalls a talk from a Yandex conference: on a simple service called Yandex.Voda it showed how, as load and integrations grow, a balancer, replication, sharding, a fallback stub (whose proper name he forgets), retries and a circuit breaker, tracing, monitoring and self-healing appear, worker count following queue depth; he is unsure all of them were in that talk.

A complex working system cannot be built in one go; a simple one that gradually grows complex can. Alexander explores the growth metaphor through a book by a mid-2000s Amazon manager, whose name he hedges: the original title used the word growing, which the Russian publisher Piter rendered literally, as in crops. That felt odd — potatoes are what you grow at the dacha — but the living-organism metaphor proved accurate: without markers set in advance, a system grows into chaos. Then the balance: the business assumes more features automatically mean more money, and engineers must stop the team hitting a dead end where features stall and reliability slips out of the agreed interval; the Google error budget helps here. Debt is counted beyond code, too — test debt, design debt where teams cannot keep up with the evolving design kit, and analytics debt where a cross-component spans five or six screens with its behaviour nowhere described as a whole. The first step is working out whom you owe and how much.

How do you put accumulated experience into a colleague's head as the team grows? Alexander's answer is to build the decisions into the pipeline: tests return green or red, security moves from a manual go-ahead to DevSecOps that scans base images and dependencies, and infrastructure is code too and can be checked the same way. Architectural characteristics follow: in Building Evolutionary Architectures by Neal Ford, Rebecca Parsons and Patrick Kua — the second edition came out in November 2022, its cover turning from grey to green — these are fitness functions. The basic case is checking dependencies and cycles, though at Google itself Chubby, Bigtable, Colossus and Spanner are deployed on top of one another, and such a system could not be bootstrapped from scratch. From Clean Architecture he takes the analysis of abstractness: a module with no interface, whose implementation everybody knows, is a communal flat where you have no room of your own and live in the kitchen — exactly what they found while refactoring the mobile bank.