Distributed Systems — Episode 4

Anton introduces himself first: over ten years of experience, languages that kept changing — C++, Java, C#, JavaScript, and Scala as the main one for the last few years — and today he acts as an architect in Tinkoff Business acquisition. Artem builds services that answer whether a product can be offered to a person at all and on what terms, and he studied a speciality about communication systems, so the chapter on communication is close to home. Before the chapter itself the participants recall Tanenbaum's other brick, the one on computer networks: by one participant's reckoning it reached a sixth edition in 2021, with new material on Wi-Fi and telecom, and it keeps gaining co-authors, since a book that size cannot stay current otherwise. The chapter opens with layers, and the OSI model surfaces at once: nobody could expand the acronym on the spot, which is fair enough, because the book itself says no protocols for it ever appeared.

The four-layer model looks more practical: a physical layer covering the first two OSI layers, a network layer with routing and IP, transport with TCP, and the application layer that absorbed the upper ones. The point of a layer is that it adds its own header so the next one knows where to send the message on; the useful payload can be small, while assembling and parsing headers is not free and sometimes hurts performance. The author has four layers too, but he pushes DNS, authorisation and distributed commit into middleware — everything we build our application on top of — and puts the operating system level below that. The participants' conclusion: what matters is not remembering whether there are seven layers or four, but understanding why layers exist and what services they give the layer above. Communication then splits into transient, where the client gets its acknowledgement and forgets the message, and persistent, where the request sits in intermediate storage until the recipient comes for it, with mail as the authors' obvious example of the second.

The diagrams use bold lines to show where the client is blocked, where an intermediate delivery acknowledgement reaches it, and where it fires a request and moves on, to get the answer through a callback or not at all. The second part of the chapter opens with history: distributed systems used to pass messages explicitly, and in the eighties it was proposed to view this as a remote procedure call — and RPC, the participants note, is used everywhere to this day. Anton walks through a local call: parameters go on the stack, control passes to the procedure, the result comes back the same way, and this is the picture to keep in mind, because it is the one that will break. Locally a large list is passed by reference rather than copied in full; there is also the copy-and-restore strategy. Remotely a pointer cannot be passed: marshalling appears, a common format, and client and server stubs generated from an interface description.

The second approach is RMI with references to remote objects; nobody could recall another language with RPC built in, and the participants have security objections to RMI: the defaults make it easy to end up with a serious vulnerability. A listener's question asks about a broken connection: it has to be handled on the client — timeouts, a missing response, retries where needed — and one participant has never met a retry mechanism wired into the stub itself and thinks the idea is poor. Retries drag in idempotency: a key that does not change on a retry so the server recognises the duplicate — except you have to land on the same server, and it has to remember the key. Fault tolerance, by their reckoning, waits in chapter eight. Anton spells out the flip side of transparency: a developer forgets the network is unreliable by definition, opens a transaction and then goes across the network — the transaction hangs, commits stall, the system runs badly. One of the hosts backs him with a book on the philosophy of design: the model in which a remote call is indistinguishable from a local one is deceptive, because the promise is not kept.

Among the RPC variations they cover the asynchronous call with a callback and multicast, where several servers are called and either all answers or the fastest one is taken; the participants recognise reads from replicas with a half-plus-one quorum and the repair of diverged values at read time, which throttles performance. The third part is about explicit messaging, needed when the process being called is unavailable. From sockets, where transport offers only listen and send, they climb to ZeroMQ with request-reply, publish-subscribe and pipeline: request-reply is the same RPC, only through a queue, and messages can be sent before the connection is even established. Then MPI: the host recalls crunching matrices on a cluster with it around 2005, with code that did not look human. What matters in MPI is the failure model: on a fault the program simply dies, because the cluster network is assumed fast and reliable, and if that means seven nines while the result is only needed by tomorrow, restarting the computation is easier.

Anton recalls a university course on parallel computing where in-machine parallelism and network exchange were two different frameworks. Next come queue managers with addressing, routing and a directory of who lives where, and behind them the broker: N participants each with its own dialect give a quadratic number of translators, whereas with a broker each needs a single mapping into its format, plus plugins, including ones that translate protocols. As a digression absent from the book they discuss Kafka: it is not a broker but a log, order holds within a partition but not within a topic, and a consumer group holds no more consumers than there are partitions. The final part, on multicast, the participants call esoteric: hardly any of them has written such a system. Artem covers the distribution tree that breaks along with a node, flooding to the nearest neighbours, and probabilistic gossip that saves the network at the cost of unreached nodes. Deletion breaks the scheme: a returning node spreads the data again, so a marker is spread instead — a certificate of death. Chapter five follows in a week.