Distributed Systems — Episode 7

The chapter has six parts: an introduction, data-centric consistency models, client-centric ones, replica management, consistency protocols, and the hosts' beloved web with its caching. The subject is scaling: fault tolerance the authors handle separately, and the club returns to it later.

Vitaly lists the reasons for going distributed — durability and availability when nodes, subnets, or whole regions fail, planned maintenance, and, for planet-scale systems, growth beyond one node and data close to the consumer.

The host liked the authors' logic, which he arranged into building blocks: performance pushes you into caching, caching brings replicas, replicas raise consistency, and the loop closes. In the model being discussed, the way out is to weaken consistency requirements, and there is no universal recipe: which consistency is needed is decided by the application developers, not the store.

The authors split the models into two families. Data-centric ones keep the focus on data and the processes that change it; client-centric ones, Vitaly explains, grew out of the mobile era, where changes come from a small number of clients, so the question narrows to what a client saw, what it changed, and what happens when it reconnects to another replica.

The host collapsed the data-centric models into one table: books list them one after another, and halfway down you lose track of which nests inside which. The notation stays constant: a time axis left to right, processes P1 to P4, writes and reads, a read returning an empty value until the update arrives.

Sequential consistency permits any reordering that preserves order within a process: what matters is not the moment of a write but a single order seen by all readers. Where three processes each write their own variable and print the other two, six operations give 720 permutations, very many of them invalid for a store.

The participants highlight a limit of sequential consistency: guarantees for individual objects do not ensure the same guarantee when those objects are used together. Hence linearizability and overlap in time: a write occupies an interval, and when intervals overlap the order depends on where the linearization point falls; the host liked that other books, as he recalls, go the other way round, from linearizability to weaker models.

Causal consistency returns to the vector clocks of an earlier chapter: a process reads a value and then writes its own, and the authors treat the second as potentially dependent on the first. Vitaly cannot find an explicit criterion of causality in the chapter — a connected read and write on the same object is taken for it — and asks viewers to check whether he missed a definition.

The link is caught only when communication goes through the same channel: a neighbour watching a match shouts about a goal before the picture arrives, and the host recalls the book the antagonist used in Back to the Future, and betting on sport.

Closer to a developer is grouping operations around acquiring exclusive or non-exclusive access to an entity — a user-supplied description of causal links, as Vitaly remarks. The model is simplified: real databases and network protocols such as NFS have more lock levels and escalate granularity, from a record to a page and a range of values and on to an object or the whole database. At the strictest isolation level a database must line transactions into a sequence, and the host was amused that a single-operation transaction collapses back into the earlier model.

Under eventual consistency, once updates stop, replicas eventually receive the latest changes and converge. The participants use familiar examples: CDNs serve images and scripts that do not always require the freshest copy, while DNS caches responses at multiple levels.

The host proposed skipping the continuous model with its three axes of deviation — numerical deviation in a value, staleness in time, and order: the order axis he called the most baffling, and knew of no application. Vitaly objected that bounding the writes made locally and not yet confirmed is what a local cache does, flushed on overflow.

Commutative operations survive any ordering: a flag that can be raised but not cleared, or counter increments and decrements requested by two users, end at the same result; add a reset or bounds such as zero and a hundred, and the outcome depends on the order again.

The host recalls how, as the Yerevan office was opening, people signed up in a shared spreadsheet, and simultaneous writes into one cell left somebody without a place: the last write won.

Client-centric models catch something else: a client reads a page, switches through a VPN, and puts a like on a different server, so the writes preceding the read must arrive before the modification. A separate pain is reading your own writes: some ten years ago, splitting writes to the master from reads on replicas broke the logic of applications that knew nothing of replication lag.

The host likes that split of flows for the scaling it buys, but names the price himself: after a transfer the fresh command must be applied to a stale read model locally, or the user will not see their money, and streaming the updates only shrinks that black magic, never removing it.

He compares the approach with the lambda architecture, whose speed layer gave an approximate delta for the day, down to estimating unique visitors with HyperLogLog. Vitaly replies that complexity is traded for lower cost, though the balance is not always positive.

On replica placement, Vitaly stresses the difficulty of measuring access times and channel utilisation globally. Contractor prices and conditions constrain the choice as well. His joke about a solution left behind by an engineer who quit yesterday underlines the gap between a neat model and operating a system.

From his trading experience he recalls delivering changes over multicast with lost packets re-requested; from the protocols, synchronous replication through a dedicated primary object and quorums, whose level, the host adds, is dialled inside a Cassandra query.

In the Tinkoff web scenario described, personalised dynamic pages stay uncached because of targeting and tests, while static files and mobile app configs go through a CDN, where a git pipeline puts files into storage and calls the refresh endpoint.