Distributed Systems — Episode 8

The host opens with bad news: Alexander is ill, they wish him a quick recovery and hope to see him for the next chapter. Denis started as a system administrator, came to Scala through Java and has written it for five years; Salikh works in the unit for opening new products, where applications land from every front end, ATM and POS terminal — "the customer's life starts with us". Early on his microphone drops out, and on returning he jokes that a fault occurred and healed itself in time. The book splits dependability into four properties: availability, readiness right now; reliability, running without failure over a long interval; safety, a failure that causes no catastrophe (the nuclear plant example); and maintainability, maintenance without losing availability. Availability and reliability differ: a single two-week outage a year still yields 96% availability. The English word dependability, the participants admit, has no neat Russian equivalent: several distinct words collapse into one word for "reliable".

Failures are the normal state of a system, and mean time to failure or repair counts only where it is clear what a failure is. The line "a fault is the part of a system that can lead to a failure" turns up, they note, in almost every chapter. Faults are transient (a magnetic pulse on the line — the resend goes through), intermittent — the hardest to diagnose, since by the time an SRE arrives no trace is left — or permanent, like a memory stick that has popped out. The crash failure is illustrated by the reset button: once on the back of the case, then moved to the front, and the author hopes it will one day move back. In an omission failure the request is dropped because there is no thread to serve it; a timing failure can run the other way too — the client is flooded with data too early and the video stream falls over. The worst class is arbitrary, or Byzantine: a process does not die, it lies, returning false instead of true. The name comes from the empire famous for intrigue and false claims, and it began, by one participant's recollection, with Albanian armies, later renamed so as to offend nobody.

Redundancy comes in three kinds: informational (a Hamming code adding bits to restore a message), temporal (retries and resends) and physical — not necessarily hardware, it can be a group of application instances; the model cited is triple modular redundancy from electrical engineering, where three voters pick the majority signal. Processes are joined into groups that look like a single whole from outside: a flat group has no single point of failure but its internal communication is complex; a hierarchical one is cheaper by the number of arrows on the diagram, yet its Achilles heel is the coordinator, and when it dies "the most entertaining part of the ballet" begins. Membership is a separate problem: either a roster coordinator that will itself go down, or everyone knowing everyone, which turns joining into a lengthy procedure. The chapter sums up as "there is no free lunch": a plain crash needs k+1 nodes, while arbitrary failures make the book demand 3k+1 — seven servers to survive an outage on two, which "sounds frightening".

Consensus by having everyone broadcast to everyone is "simple, clear, obvious and wrong", consensus in a kindergarten sandpit. The practical example is Raft, and one participant was surprised it comes before Paxos, historically first but complex and little used. In Raft every server keeps a log of commands, the node with the freshest log becomes leader, and a follower relays a request to the leader in full — you think you are not hammering the coordinator, yet your requests fly there anyway. HashiCorp Vault, the secrets store, recently gained its own Raft; before that it needed an external system such as ZooKeeper or Consul, itself built on Raft, and in Kafka, by another participant's older recollection, there were plans to drop the external ZooKeeper for an in-house one. Paxos, by one participant's reckoning, is an island somewhere in the Adriatic off Greece, and an algorithm described as the sittings of a loosely democratic assembly: the confusion spawned interpreters, joined some eight years later by the author himself with a second paper, and implementations never reproduced the ideal. Two limits close the part: enormous cost, and the CAP theorem, of whose three properties you pick two.

Communication channels suffer the same classes of error, and the job is to mask them. TCP covers retransmission, but a broken connection is plainly visible to the client, which usually re-establishes it at the application level; RPC tries to hide the network entirely and still forces you to handle its exceptions. A resend requires the server to tell duplicates apart: idempotent calls, client-side message numbering, a repeat flag — and among the delivery semantics one participant singles out a fourth that guarantees nothing. For a group the book separates receipt from delivery, saves traffic with negative acknowledgements, and builds ordering up from none through FIFO and causal on vector clocks to total, the strictest. Two-phase commit breaks when the coordinator dies in the waiting state before broadcasting its decision: the nodes must wait for it to return. Three-phase commit adds a pre-commit phase, on the strength of which the participants settle the transaction themselves.

Recovery is either forward or backward. Forward recovery requires knowing every error in advance, so it is almost never met; backward recovery knows nothing about the process and simply takes checkpoints, so it applies everywhere. Yet even a successful rollback does not fix everything: if the ATMs have dispensed the cash, the system returns to the past and the money does not. A coordinated checkpoint uses a two-phase-commit protocol; an independent one is cheaper but risks a cascading rollback to zero; between checkpoints a message log is kept, and pessimistic logging forbids answering until receipt is recorded. Hence Denis's objection: the book keeps proposing to repeat the command, while in the real world you cannot send the money twice. In his own practice the payment went through, the timeout fired after a minute, the system reported failure, and the customer got a text saying the money had left — untangled by hand, outside the system, against the register. Along the way they compare editions: Raft was already in the third, while blockchain was added to almost every chapter.