Designing Reliable Systems — Is It Worth the Effort?

Two terms are separated up front. Availability is the share of time a system is reachable for users; reliability is the probability that the system meets the required level of performance. The author immediately concedes that the industry uses the words loosely and often interchangeably. Three reasons for forgetting follow: invisibility, since reliability, like security, is mostly invisible while things go well; assessment, since chasing perfect reliability is impractical and calls for risk management and a price tag on bad events; and evolution, since a system inevitably grows more complex as features and scale are added.

The level of risk is chosen per class of application rather than for the company as a whole. The white paper Deployment Archetypes for Cloud Applications supplies the split: business critical systems with direct business impact and hard availability and latency requirements; line of business applications such as ETL/ELT jobs and CI/CD pipelines, where execution speed matters; and internal applications such as HR and office automation. The author notes that companies usually make this classification far more granular to fit their own SLA strategy.

The definitions come from Google's SRE book. An SLI is a quantitative measure of some aspect of the service delivered: latency, error rate, throughput, availability. An SLO is a target value or range for an SLI, written either as SLI below target or as a corridor between a lower and an upper bound. An SLA is a contract with users that spells out the consequences of missing an SLO. The practical test is simple: if a breach carries no explicit consequences, you have an SLO, not an SLA.

Monitoring splits into white-box, which exposes the internal state of the system, and black-box, which reflects what the user perceives. For ready-made metric sets the author points to Google's Four Golden Signals and Tom Wilkie's RED Method. Both serve the purpose SLOs were introduced for: noticing a broken promise before the user reports it.

Four concepts carry the design part. Fault domains are sets of infrastructure components that fail as one, which argues for replication across zones and regions, load balancing, and short startup and shutdown times. Sharding spreads applications and data across those domains, incremental rollback-friendly deployments keep production changes reversible, and dependency management accounts for the availability and failure models of everything you call. Data gets its own trio of durability, availability (asynchronous replication gives eventual consistency, synchronous gives strong consistency) and backups, while RPO and RTO targets drive the deployment model. The warning is blunt: adding availability as a feature later can mean re-architecting or fully rewriting the application.

The pattern catalogue is credited to Alexander Krivoshchekov of Yandex Go. Retries apply only to idempotent requests and should use exponential backoff with a randomised interval. Deadlines are client-side timeouts coordinated across the whole call chain rather than set per hop. Rate limiting relies on token bucket, leaky bucket or sliding window counters. A circuit breaker cuts off calls to a broken provider once an error threshold is crossed. A rich client moves computation to the client, with the caveat that applying it everywhere over-complicates the system, and a dummy offers a stripped-down emergency mode. The closing table of archetypes makes the trade explicit: moving to the right raises reliability and cost together.