Distributed Systems — Episode 2

Alexey introduces himself: eighteen months in the investment area, rolling out architectural practices and building high-load systems that give clients access to world markets; before that came infrastructure, software development and a Twitch-style video streaming product with paid earnings that never took off. Sergey Ivanov has been designing for over twenty years: online casinos back when that was not yet illegal, banking systems with a complex domain, infrastructure they assembled themselves back when clouds were still unknown; since 2019 he has run an architecture conference and more than a year ago helped found an architecture association, where deciding what to call its practising consultant ran into one of the main problems of development. The participants praise the book for introducing a term in the middle of a narrative and citing the work where it first appeared. The flip side: pulling a definition out of that context is hard, and one participant admits his notes end up simply copying the text, so concentrated is every page.

Architecture, for the authors, is what components a system consists of, how they connect and interact; next to it stand the middle layer separating applications from the platforms beneath and system architecture, the deployment on real machines — the part, one of the hosts notes, that system design interviews rarely reach. Layers are a logical split, tiers a physical one, and the definition works by opposition; the same components can live inside one process if the application is assembled from loosely coupled pieces with either in-process or network transport. A layer is described by the trio of interface, service and protocol, with network stacks as the example. The upside: implementations are interchangeable and layers test more easily than one huge construction. The downside is the price of crossing boundaries — headers and data transformations across ten layers can cost more than the useful work, hence header compression, buffering, and attempts such as HTTP/2 and QUIC, plus the administrative price of maintaining every protocol.

The classic three-level scheme gets skipped — everyone started there — and the conversation moves to service-oriented styles, where the authors mixed object-based architectures, microservices and REST. Remote method invocation is recalled without fondness: you call a local method and it executes on another machine; building a test environment for that is next to impossible, latency was constant, and the industry moved on to message passing quickly. One participant hedges that the approach itself may not be to blame: the middleware of the day was probably written badly, and done well the idea could have earned its place. The problem, the participants argue, is false guarantees: middleware hides both complexity and the idempotency of calls, so a developer writes code as if the process crashes as a whole, yet waits an order of magnitude longer and catches errors where none were expected. The mental model does not change between a local and a remote call, so you cannot even tune the application properly: you do not know where it stands.

One of the hosts ties this to the cognitive complexity described in a book on the philosophy of software design he is reading in parallel: complexity hidden under guarantees breaks through at the worst spot — you walk across beautiful thin ice, fall in, the current carries you away, whereas knowing about the depth you would have built a bridge. Microservices are traced back to object-oriented design: composing a large service out of small ones, where modularity and logical boundaries are the key; at first people split systems any which way, and when it got painful, domain-driven design came to the rescue. The book gives them a single block, but a solid one: there is no agreed size, what matters is separate processes, possibly on different machines, and genuine independence. There is a complaint too — REST is singled out as its own style even though a microservice happily exposes both RPC and REST; REST itself is praised for a human-readable set of resources and operations and for leaving state with the client. Client-side composition follows from the same shift: the main consumer became a person, not a computer.

The coupling table sorts interaction along the referential and the temporal axis: a direct call is coupled on both, a named recipient's mailbox decouples time, an event bus decouples the reference, and the most decoupled corner is a shared data space, illustrated by a file share where colleagues drop files and look in once a day. Subscriptions come by topic, by key and by content: the book subscribes to events with coordinates inside a geographic area, while the working example comes from investments — a portfolio-updated event, with the service deciding whether bonds or shares concern it. That brings up a technical limit of popular brokers: a topic per client is physically impossible with three million of them, so applications read the whole stream and check every event. Middleware is described as the operating system of a distributed system: resource management, communication between applications, security and accounting for consumed resources. Middleware is extended with wrappers and interceptors: wrappers between every pair of applications produce a fully connected graph nobody wants to write, a broker collapses the picture into a star, and the modern interception example is a service mesh, which takes over secure interaction and service discovery and can do mutual authentication with trusted keys.

Among system architectures the authors separate a vertical split by function from a horizontal one, where nodes are identical but hold different parts of the data — much like splitting columns versus sharding in databases. Three tiers are the classic, yet real systems reach eight or nine: an investment client's request passes a million applications and transports on its way to the exchange. Peer-to-peer networks, where a node is client and server at once, split into structured ones, whose topology needs an administrator, and unstructured ones, which live with unstable membership and spread information by gossip. The last third goes at a run: the cloud offers a pool of resources and elasticity, yet cost, guarantees and regional capacity stay open questions, so services with a vanilla installation are preferable; edge pushes work towards the client, where a hundred milliseconds already matters for a driverless car, and Amazon will ship racks to a customer's own site; on blockchain the participants stress immutability, recalling how mutable chunks in Google File System forced distributed transactions while Colossus made them immutable.