Architecture Governance

Alexander welcomes Pavel Lakosnikov, by his own reckoning in his ninth year at Avito, and explains why: he does something similar at T-Bank, and judging by their talks the Avito team has built an interesting process. They start with the Russian notion of exposure, a word Pavel finds vague yet impossible to deny. Alexander compares it to photography: to grasp what good means, theory is not enough: you look at the best authors' work and work out why their shots land. Operations gives the same feedback: a system is painful to change, or drags you out of bed at night. As an engineer, Pavel is bothered that none of it reduces to a metric — you can hardly count the projects you have seen, or ruined. Avito does have an architecture section in hiring, though: which large projects you took part in, what your role was, which hard decisions you made. Those who went through that school answer with their eyes alight; others stop at a memorised phrase.

When Pavel joined, Avito was one big fat monolith; he built features next to the core and touched authorisation a little. Roughly a year later the company decided to move to microservices. A legend went round about the largest unsharded database — refuted a couple of years later, and Pavel no longer remembers the figures. There was plenty of logic in stored procedures, but mostly it was data access logic. There was also a point of no return: a date was named after which the database would simply not come back up, and as it approached it went down now and then, with DBAs rescuing it by micro-tuning configs. A separate source of cringe is a service that took data from three services and returned a nicely decorated price: only PHP code could render it, and the teams that suddenly needed it had lost their PHP expertise. Alexander turns the picture round: these are not bad decisions by predecessors but a growth problem, the problem of a happy company.

The second thread is how the processes changed. A monolith release was assembled about once a day: a separate branch with all the fresh changes, long test runs — half an hour in good weather, five or six hours in bad — and a group of gurus deciding at the end it seemed to work. That alone acted as a decent quality gate: big failures were rare, and on-call amounted to a phone call waking everyone. With microservices, releases became frequent, incident management appeared, and behind it a question: incidents do get fixed, but is anything actually getting better? That is how a dedicated team appeared, and Pavel entered architecture through reliability. Is it like SRE at Google, Alexander asks. Conceptually a subset, Pavel answers: theirs was more a central team that knew a lot about everything because it took part in the break-up, but never part of product development. They also caught basic mistakes such as opening a transaction and then calling a neighbouring service.

For measuring the internal reliability of a complex product there was no ready answer: Amazon publishes an external SLA for S3, but how to judge the reliability of the shop as a whole and trace it down through the systems they had to invent themselves. Avito now separates technical and product reliability: product reliability is hard for a platform team to watch, so they started from sets of user scenarios — open, add to basket, buy, order delivery. Product owners describe scenarios and sort them into criticality bands; tooling turns the descriptions into a graph of service links with technical metrics hung on it. As a bonus, component criticality is no longer assigned by expert opinion: if your service lands on a scenario with no weak links, congratulations, you are critical now and hold three nines. Some priority arguments disappear with it. Alexander has a comparable catalogue of services with fixed SLAs and criticality, but tracing is not threaded through every system, so a live chain can break in the middle.

Alexander likes the Russian word for architecture supervision far less than architecture governance: supervision hints that somebody must watch over you, while governance in the books is a management model of standards and policies, of crystallised practice. He translates that into architecture as raising the baseline: set a quality bar you may not fall below, then lift it gradually. To newcomers who picture the governance routine as walking in and fixing everything, Pavel explains that it does not scale: you create conditions in which the right behaviour pays better than the wrong one. So a rule always arrives with tooling: that is how the standardisation of RPC interactions went — by Pavel's estimate around 99% of internal calls now follow their protocol — and how a contract system between front and back end is being rolled out, with screws to be loosened once basic adoption is there. The other half of the job is consulting: have you thought about security, about personal data transfer, about the load you will put on a service that runs on two miserable pods?

Metrics are harder. Classic cyclomatic complexity and module isolation describe a single application rather than a microservice landscape, so Pavel's team is testing inter-service coupling and cohesion, taken from papers and checked with thought experiments: add a link and the number does not move, remove one and it improves. Alexander counters with an example: knowing the formula, a spider-like service can be split into an inbound and an outbound one with a single thin link between them — coupling drops on the diagram while the data passed does not change. Shortly before the recording, after a very similar chat discussion, someone described exactly that scheme to Pavel from real life. Their non-functional requirements tool has the same trap: a team that breaches its own gates simply widens its own tolerances, and governance then has to work out what caused it. In closing, Pavel says no ready recipe exists, and argues for getting involved in building solutions — that, he says, grew his own expertise.