Reliability: Never Roll Out to All of Prod at Once

The guest introduces himself as a staff engineer on the Databases team led by Alexey Svetlichny, who had a separate episode recorded shortly before. By his own reckoning he knows one database well — Cassandra; the rest, from Postgres to the vector store Qdrant and caches, sits at an average DBA level, and most of his time goes into turning databases into a platform service rather than engineering. The work has become managerial and architectural, with less hands-on thrill, though with heavy operations and few people, hands are still needed. He was born in Izhevsk and grew up as the Soviet Union fell apart, when the only entertainment was the street and the library; the passion for reading stayed for life, though he reads fiction rather than technical books, and donates to contemporary authors. In 1998 he entered a provincial university — there was no money for Moscow — where you could bring textbooks to an exam but had to answer from a blank sheet, and one answer could run to six hours.

That same year George Soros was opening internet centres in the Russian provinces, and in the university's new building the guest first saw Linux: the incomprehensible running lines turned out to be magic, and he has lived on Debian ever since, giving way only on the corporate laptop with Ubuntu that security requires. His career came in pieces: olympiad coaching at his old school, a pharmacy chain, a university department, a research outfit, and in 2003 the state power grid, where branches across the republic worked as thin clients over SSH against a server running an old FoxPro application under DOS, which was too convenient to replace. Around 2005, in telecom, he built fault-tolerant mail, hosting and DNS on eight servers: LXC and BGP gave him containers long before Docker, and clients never noticed a failed link or server. He changed jobs when things got boring and the market offered several times the pay.

Odnoklassniki happened differently: after a conflict inside Rostelecom he snapped, answered a vacancy, his future manager rang from Latvia, and buying a stranger tickets and a hotel room off one call struck him as wild. He calls that interview one of the easiest of his life — a conversation about what he had done, where he had been wrong and how he fixed it; with the then CTO they talked about firearms, the guest being a ranked competitive shooter. It was 2012, remote work was frowned upon, and he moved to Moscow. Fewer than a hundred developers and admins ran a service bigger in traffic than his current company's; on duty you answered for everything and could push back on any CTO or ring the head of the parent company. A newcomer spent two weeks devouring documentation — nobody called it runbooks, but that is what it was. There he met Cassandra, disliking it at first, and became the specialist on Voldemort, a distributed database built on Lamport clocks; he moved the pairing off Tarantool, which behaved badly in production, onto Cassandra.

In 2015 the site went down for three days: changes went to all of production at once, security switched auditing on in bash, and on the operating system of the day — SUSE 10, as the guest recalls — a bug hung the shell, while rebooting did not help because bash started at boot too; the fleet came back partly by hand; he lost three nights of sleep and missed his daughter's fifth birthday. After that the whole company turned towards reliability: some did not survive the turn, but later a data centre could be switched off unnoticed, and a fire in one woke three or four engineers rather than all. Alexander adds that outages of that scale are called black internally: they launch a year of reliability and standards such as phased rollout, which would have shown the problem on the first machines. His other point of pride is moving a data centre in two months as the project manager for infrastructure: rack and subnet maps, disks that did not survive the trip, a 50-millisecond limit that made one cloud coordinator immovable, and in the end two downtimes of five to seven minutes with no data lost.

At T-Bank the conversation with a vice-president did not click, while the Cassandra section turned into an hour of him asking the questions: why those buffers, why built that way. He calls the talk with the head of the reliability centre the best interview of his life — calm, no visible pressure, key points written down and questions asked about them; he came out wrung dry but happy. Troubleshooting felt easy right after an incident review he had been through, and in the coding section his one-line solution with two pipes was rejected, so he unrolled it into loops, arguing that pipes are the same as libraries; Alexander agrees. Both settle on difficulty needing to be variable, like a games master in DnD, yet hard to formalise; the guest, recently made lead of that section, is building such a scalable task. He also picked the hiring process apart: figures from the recruiter and accounting disagreed, and a contractor's app showed a newcomer every candidate's data at the offer stage.

On day one he asked for the list of clusters and incidents, and what he saw made his hair stand on end: a great deal was wrong, few designed data schemas for Cassandra well, and Cassandra is an ordinary JVM application where, with default settings, a 32 GB heap leaves it less memory than 27. He fixed it two ways, technically and by teaching: meeting teams, explaining the basics, and starting Cassandra meetups that have since stalled. Six months in came the first challenge — carving data out of a database that was, he says, on Oracle into Cassandra: the model took him and a strong engineer a long time, the cluster was tested by killing nodes and a whole data centre, and the migration ran with zero downtime, while the harder part was securing hardware. Cassandra as a service grew out of that: several runtimes, any database wrapped in pods, Qdrant built to the same template, and Chebotko diagrams drawn with large clients. His closing advice is to read, fiction included, to learn to skip the padding and spot it in documentation.