Architecture at the Start: Preparing for Success

Alexander opens by arguing with the question itself: architecture does not always need to be made reliable. It helps to sort systems into classes — mission critical ones, where an outage can cost a licence, business critical ones, where it costs money, and internal tools, where colleagues simply wait for the repair. The class decides how much a company will spend, because reliability means duplication that genuinely costs money, a higher class of hardware with its own guarantees, and engineers who can implement the patterns. He says he has often met the opposite: people had never heard of approaches inherited from networking, such as load balancers, had read no books about networks, had given little thought to distributed systems, and built what they built. Hence his first example: the outage is fixed, but everything queued up during the downtime arrives at once and knocks the service over again. That is a metastable state, and by his account research papers are devoted to it.

Oleg picks the thread up: a system has to be reasonable first of all. You can read a mountain of literature and work through every failure mode, but by the time all of it is woven in, nobody may need the system any more. His analogy is a small home bakery: securing it by the book would need a ten-tonne bank vault door opened by two people and costing more than a century of the shop's turnover, whereas ordinary glass a hooligan can break is a risk you accept. He has his own example too: a project theoretically designed for a billion people ended up with an audience of fifty thousand, and the overhead of extreme reliability never paid off. Filipp adds that reliability is not only about the system running but about it running the way it should: records vanish unnoticed, money hangs somewhere, replication keeps ticking and everything looks fine from outside. He clarifies that he means reconciling data by two independent routes rather than monitoring as such.

A chat question about what reliability is measured in leads the host to availability and a nines calculator. Oleg lists the classic metrics: mean time between incidents, time to alert the team, time to recovery, an assessment of business impact and the cost of downtime that follows from it, measured not for the numbers but to stop failures repeating. Filipp objects that an SLA is a resulting metric rather than a design one: design starts from RTO and RPO, and for data from integrity and the ability to notice it has been broken. A minute of downtime at peak and a minute at night that a user sees only as a spinner weigh the same formally, yet differ by orders of magnitude in real losses. He separates the SLA from the SLO: the first is a contractual obligation with a penalty, the second is what the team keeps in monitoring and what wakes the on-call engineer when a promised 120 milliseconds at a high percentile is missed.

Alexander shows how this works at T-Bank: an internal tool holds the contracts for user-facing services, down to a promise that 99.9% of requests will complete successfully; compliance is computed from the shared observability platform, and incidents are raised, linked and marked with the root one. The business gets a report against those contracts rather than a lecture on RTO and RPO. He also recalls a talk on how Google separated platform incidents from user ones: the platform does not answer for a workload a customer deployed badly. Oleg counters that everyone gets paged regardless of who erred, and the fix ends up cross-team. His own company, he says, runs an observability platform handling several billion events per second on fairly modest hardware, plus an internal reliability service where a component holds its tier only if a set number of action items is closed after an incident. A participant from smaller companies pushes back: with twenty or even two hundred services a Confluence page is enough, and communication matters more than tooling.

The most important instrument, in Oleg's view, is the drill: however eloquently you describe your conduct in a crisis, it is better to reproduce the failure under control, confirm you do not survive it, put the components back quickly and repeat. Otherwise you find that behind that ten-tonne door the hinges fail on the second opening and the wall falls out. One of the participants recalls shutting down a PostgreSQL master node to watch the switch to replicas, and the regular drills at Yandex where a whole data centre was taken offline: the first rounds surfaced a great deal, including services running as a single instance, while six months later teams treated it as routine. Alexander advises starting with tabletop drills, then disabling small things, and only then going wider; chaos engineering he treats as the next step, a continuous practice against people forgetting where to run when incidents are rare. A seventy-page report one expert read some five years ago ended with a warning: if production already supplies enough problems, first learn to absorb them invisibly.

Asked what an engineer should study, Filipp suggests collecting your own bruises and meeting ninety per cent of the standard problems in practice, leaving patterns for later. Oleg adds: return to goal setting, remember the security people, and document the project no less than you test it, because that debt is worse than development debt. On testing he points to a talk by his colleague Vitaly Gridnev about how their database is tested: by the statistics quoted there, around 80% of problems are caught by writing tests. Alexander steers the talk back to the building blocks: installing Kubernetes with a replication factor of three is not magic, you must know what the underlying nodes guarantee — Google built GFS, the ancestor of HDFS, on cheap servers precisely because it did. Reliability, he says, is emergent: there is no chunk of code where it was implemented. Filipp closes: everyone always lies — vendors, the business, developers and operations — so dig one level deeper: test the database yourself or read what Jepsen writes about it. Audience questions add the economics: manual workarounds grow expensive as a product grows, and a packaged product must count its total cost of ownership and leave a dead horse promptly.