SRE: Hand On-Call to the Developers

Salikh introduces himself: some eighteen years in IT, a fourth year at the company, currently a chief engineer in the unit that handles the launch of new products. A computer appeared at home early, around 1996 by his reckoning, which was early for Kazan; at fourteen a friend who worked as a sysadmin drew him not into programming but into administration, and the first system besides Windows was FreeBSD, which he broke constantly. When fast internet reached the city, the computer clubs were building a shared gaming network, and he talked his way into a job there: access to the servers came straight away, production was real, and after one careless command he had to drive to the data centre on a hastily arranged pass. He studied to be an aircraft designer, with many computer science subjects missing from the curriculum, yet code did the integrals for him, and some lecturers happily took the sources instead of a written-up report.

His first paid job came in the second year, from a five-year-old article that ended with an ad for a PHP programmer in Kazan; on arrival he was told they needed front-end markup people, and he agreed. He spent more than eight years there: markup, PHP, mobile development — in 2012 he ran macOS in VMware because the company owned no Mac at all — then infrastructure, group lead and finally head of development with up to sixteen people, effectively the CTO of a small company. As a group lead he owned tactics three or four months ahead; as a head of development, strategy, hiring and value-delivery metrics. In 2020 the pandemic hit the company's advertising business, Salikh went to the market through friends and spent a year and a half at a startup, a social network around medicine, where he built the delivery process quickly and grew bored.

While still at the startup he took an SRE course taught by Dima Maslennikov — the course was run by the large company that invited him to join once he had finished it — and realised the role suited him: there is infrastructure, there is code, and the team's influence rests not on direct authority but on communication and the ability to convince other people's developers to make an application more reliable. Hiring took about six months and, by his count, five sections: four technical ones — basic programming, algorithms, system design and troubleshooting — plus a fit interview; Alexander recalls that few candidates did well at system design and the section was later dropped, while Salikh had one of the best results, which he credits to the startup. He joined the large company as a senior engineer deliberately: he knew nothing about fintech and wanted to understand it from the bottom up. He grew to lead engineer fast by picking up postponed problems — traffic skews and database performance that suffered not from bad databases but from how they were used.

The team he joined is seven engineers plus a manager who came up as an engineer himself and still works incidents; they are on call around the clock and own the availability of the whole unit. The origination unit is over six hundred people in six departments: the first application forms for any product and everything that follows, including creating and checking customers and scoring risk. Salikh tallied the numbers on purpose: about 280 services, roughly forty per engineer, more than two hundred Kubernetes deployments against three hundred and fifty half a year ago, 210 PostgreSQL databases, around three hundred topics, plus a fleet of virtual machines and bare metal he left off the slide. Some systems will never move to Kubernetes — servers with four terabytes of RAM, for instance, which Alexander guesses belong to anti-fraud. The unit has 232 developers, roughly 33 per SRE, or 3% against the under-10% suggested in Google's book.

Once the service count passed three hundred, the team stopped keeping up: it had turned into a support team that only answers questions and fixes production, engineers were burning out, and no time was left for its own products and automation. Two years ago the team agreed on a change and sold it internally: daytime on-call moved into the development teams. The effect came in two parts. First, the SREs went back to what they should be doing — reliability and availability, disaster recovery plans and drills, where working out how to break a service without leaving traces is a science of its own, and observability over two or three hundred services. Second, developers gained awareness: they see how their changes break production, they monitor their own applications without fear, they switch off useless alerts and fix metrics themselves, which used to be sold to their managers as tech debt. Both time to market and recovery time improve: when an SLO is breached the team now writes that it is already rolling the release back. Alexander calls this a closed feedback loop, and Salikh adds that without it people simply stop feeling the pain.

A separate part of the job is reviewing RFCs and architecture: Salikh counts it among his achievements that SREs are always in the review now, bringing a pile of uncomfortable questions — what happens if this breaks, what are your timeouts. He recalls an architecture review of a system stretched across data centres: asked what they would choose under a network partition, consistency or availability, two engineers from the same team answered differently; they were sent to align offline and came back with the system properly laid out. Alexander adds that such questions repair the specific design and widen people's thinking, so they account for them by themselves later. The point of the early stage is not merely that reliability gets considered: metrics are easier to include and sell while budget for them exists, and the required reliability class is an architecturally significant requirement backed by RPO and RTO. The participants then look at where architecture does harm: retries at every layer add up to a storm of requests, while redundancy in fintech runs into regulatory requirements and the price of consistency.