Distributed Systems — Episode 5

Synchronization means doing the right task at the right time — the hosts open with that definition from the book. The first example is a build driven by make: one machine holds the timestamped source, another runs the compiler, the clocks drift apart, and an object file nobody verified ends up in the build. The authors then walk back through the history of measuring time: a turn of the Earth is a day, an orbit a year, an ordinary second is an imprecise oscillator, so atomic clocks arrived in the mid-twentieth century and the drift is absorbed by inserted seconds. The participants separate internal synchronization, which pulls a group's clocks together, from external synchronization against a reference: if every node is aligned to within alpha, the spread inside the group is already two alphas. NTP is walked through on a picture with four timestamps, T0 to T3: it buffers eight pairs of values and takes the minimal delay as its best estimate.

The strata get compared to DNS: expensive precise clocks on top, nodes wired directly to them below, then levels reached over the network, where accuracy falls to tens and sometimes hundreds of milliseconds. Time may not run backwards either: instead of jumping, the ticks are stretched, so eleven seconds are counted out where ten were expected, or compilation suffers again. One participant admits he worked at a hosting company but never looked inside NTP, and was surprised how plainly it ships local marks around. The topic drifts into physics: a participant recalls a book about the Royal Institution's Christmas lectures, held for almost two hundred years, and the experiment with clocks flown on a plane — they ran ahead rather than behind, because the gravitational correction outweighed the one for speed. Back to engineering: Google's TrueTime in Spanner mixes GPS with atomic clocks for a tighter estimate, while in wireless networks, where battery is precious and the timestamp is taken at the network level, the familiar NTP-style algorithm will not fly.

Logical clocks start from Lamport's idea: what matters is not the wall-clock reading but that A precedes B, and if C follows B then C follows A. The processes tick at different rates — six, eight, and ten units — and a message that left P2 at 64 arrives at P1 at 54, which cannot happen; the rule makes the receiver push its counter above the message's mark. Causality is explained by a banking example: replicas in San Francisco and New York, a client adding a hundred dollars to an account holding a thousand while the bank posts deposit interest at the same moment — the order decides the balance. Vector clocks add a position per process and make conflicts easy to spot, yet capture only potential causality: something said in the same room outside the channel never enters the vector, and middleware cannot know whether the application used what it received. One participant's favourite example: two people in a chat read the match score from different replicas, so one writes about a great goal that the other cannot see yet.

Exclusive locks begin with the centralized option: a process takes a lock like a semaphore in local programming, two or three messages are enough, but scaling is hard. Distributed algorithms make processes negotiate by messages and the traffic grows; in a ring the token grants the right to lock, but the longer the circle the longer the wait, and recovery after a lost node is the hardest part — which is why, the participants note, the plain centralized approach usually wins in practice. One of the participants recalls that Google decided not to build coordination into every system and made Chubby instead, a centralized lock service the others are built on. From the outside it is a simple interface; inside, a distributed fault-tolerant service with elections of its own, shardable to a lock server per domain, ten or twenty per system. Token ring comes up as networking antiquity: a protocol of a few megabits per second, invented, by one participant's reckoning, in the mid-eighties, which lost to switched networks.

Leader election in the book is reduced to two extremes, the bully and the ring algorithm: in the first the highest available identifier wins, in the second a message travels the circle collecting candidates. Between them, a participant adds, lie many optimizations, such as naming a successor in advance — the way a king announces an heir while alive. The shared trouble is a split: each part would find its own leader, so a majority quorum is required, and followers learn the coordinator is gone through a separate signalling channel. Raft gets more detail: it does not solve Byzantine faults, membership is known, a transaction log sits inside, a node is a follower, candidate, or leader, and the re-election timers are offset. To a client Raft looks like atomic storage: no answer until the write has spread across the live nodes, and in a three-against-two split the minority is left without a quorum. The book and Consul's guidance agree on five to seven nodes; hundreds would mean extra traffic and constant re-elections, and changing membership required an intermediate configuration in recent versions.

Where no quorum can be gathered, Ilya turns to blockchain: in proof of work contenders search for a string that makes the hash start with five or ten zeros, and difficulty is tuned by that number; one of the participants remarks that those minutes of searching are why you cannot pay with cryptocurrency the way you tap a card. In proof of stake a public random function picks the leader by tokens — and the participants say frankly that they did not understand how it works or why it could not be brute-forced. Gossip is better read not as office rumour but as an epidemic: a random encounter passes the information on; storing every neighbour across a thousand nodes is expensive, so a partial view is kept and refreshed by exchanges. In event matching, subscriptions are either replicated to every node or events broadcast to all of them; in selective routing the filters live on the brokers, while gossip forms circles of shared interest — like this book club. The finale is positioning: GPS with atomic clocks aboard the satellites, Wi-Fi by signal strength where no satellite can be caught, and Daniil's story of reading device identifiers to find what a hundred cars on the road had in common, so one advert could reach them all.