Designing Reliable Systems

The starting point separates availability, the share of time a system is reachable for users, from reliability, the probability of meeting the required level of performance, with an admission that the two are routinely conflated in practice. Reliability is forgotten because it is invisible while everything works, because perfect reliability is impractical to assess and so demands risk management and a price for bad events, and because systems evolve: growing complexity turns small changes into large failures.

Risk is then made measurable. Applications are sorted into business critical, line of business such as ETL/ELT jobs and CI/CD pipelines, and internal ones such as HR systems, while the promise is pinned down through an SLI as a quantitative measure of the service, an SLO as a target value or range, and an SLA as a contract with consequences. With no consequences attached, it is still only an SLO. Watching all this falls to white-box and black-box monitoring, with the Four Golden Signals and the RED Method as ready-made metric sets.

The design part rests on four concepts: fault domains with replication across zones and regions, sharding of applications and data, incremental rollback-friendly deployments, and dependency management. Data adds durability, availability through synchronous or asynchronous replication, and backups, while RPO and RTO targets directly determine the deployment model. The talk stresses that availability retrofitted as a feature may require re-architecting the application or rewriting it outright.

The pattern catalogue is drawn from Alexander Krivoshchekov's talk at Yandex Go: retries for idempotent requests with exponential backoff, deadlines coordinated across the call chain, rate limiting via token bucket, leaky bucket and sliding window counters, a circuit breaker triggered by an error threshold, rich clients, and a dummy emergency mode. In the table of deployment archetypes, moving to the right raises reliability and cost together, and the choice is often dictated by legal requirements on where data lives or by hard latency budgets.

The operations half leans on the book Accelerate and the four DORA metrics: delivery lead time and deployment frequency for tempo, mean time to recover and change fail rate for stability. Behind them sits an internal developer platform, which at Tinkoff is Spirit: version control; pipelines with Allure test management, a quality gate, Cosmos for load testing and Danger for fitness functions; an XaaS layer with managed Kubernetes, Postgres, Kafka, Cassandra and S3; and operational systems including Sage for observability, offered to external users too, SLAser for SLA tracking and OMG for incident management. So many roles are involved that the author writes it as Dev…Sec…Data…WTF…Ops and reaches for Roy Osherove's pipeline-driven organisation.

The culture section opens with Google's Project Aristotle: 180 teams, 115 of them engineering and 65 from sales, ranging from 3 to 50 people, studied through 250 statements from the annual engagement survey and hundreds of double-blind interviews. The five success factors, in order, are psychological safety, dependability, structure and clarity, meaning of work and impact of work; team size was not statistically significant, although other research favours teams under ten. Ron Westrum's 2004 typology follows: pathological cultures look for a scapegoat, bureaucratic ones seek justice, generative ones dig for the underlying system problem, and Westrum himself cautions that this is correlation, not causation. Google's postmortem culture closes the argument.