Databases and Platforms: Logic in the Database Hits the Hiring Ceiling

Alexander opens with how they met: five or six years ago they shared a hotel room at a conference in Tiraspol, where both spoke at the local house of culture — Kostya on sagas, Alexander on applications and Kubernetes — while much of the audience had come as to a weekend cultural event, with children and no ties to IT. Today Kostya directs the infrastructure support services department at X5 Digital: his remit covers infrastructure, the platform teams, competence centres and guilds, the automation platform, the data platform, the project office and the pricing product area, while X5 Digital builds delivery services for the X5 retail chains and integrations with outside storefronts and delivery companies. Then he rewinds to the start: Tula, the polytechnic university, a degree in comprehensive information security and a great deal of mathematics — a school, as he remembers it, in learning fast and solving problems against the clock.

Somewhere around his second year a coursemate pulled him into a research institute: no pay, but there were databases and warehouse accounting systems. The "toy" job stretched to five years of trips across the country, hardware-and-software rollouts and state acceptance tests; in a book he read there the author described how a broken query halts production and the database specialist is called in to fix it, and Kostya decided that was a splendid job — the ambulance-with-flashing-lights image is Alexander's. Import substitution added a line about PostgreSQL to his CV, and when the project ended and the Moscow arithmetic did not add up on money, a company in Tula invited him to interview with the promise of a Moscow salary: they handed him a flash drive with a database and asked him to speed up a query, move the data into PostgreSQL and speed it up again. He had no expertise, and the job meant moving several hundred gigabytes of production without stopping the business: a year of work, a Microsoft stack, PostgreSQL including on Windows, basic questions on a specialist forum where the volumes surprised people, and Saturday-night trips to the office, because laptops were not allowed home.

The move was decided together with his wife, and on the last day of his holiday Kostya was invited to interview at Avito: a beautiful office, people with a spark in their eyes, and three hours of interview: half an hour of questions, then Misha on battery-backed controllers, tuning database and file-system settings, and logical replication. Around 2014 about five people, he thinks, worked on the databases; all the business logic had historically been written inside the database — materialisation, denormalisation, tables kept in memory — while trigger-based logical replication spread data to other nodes, to the warehouse and to search indexes. There were around three clusters, as he recalls, and that was exactly when they started splitting them by domain. Listing moderation, phone verification and payment for listings were written by hand, and millions of people saw the result immediately; in under two years Kostya became a team lead. But the model did not scale: experts like that are almost impossible to hire, the business grew faster, and people left for new challenges.

Next came breaking up the monolith and moving logic out of the database: they took not the hardest service but the second, moved people into cross-functional teams, and held a load of around 20K rps with sagas for distributed transactions and a Tarantool cache, because replica lag was unacceptable. It ended in platform teams, databases as a service and a data bus: the stack was not Java, Kafka fitted best, they studied how to stretch it across data centres and talked to Confluent while working out what was being sold to them — then wrote their own. The team benchmarked itself against outside experts: people from Zalando, the authors of Patroni, came by, and clearly both were on roughly the same path. Their own Patroni equivalent also ran in production, but they later chose the ready-made tool that had pulled ahead.

Another thread of the conversation is the community. In Tula Kostya did not know conferences existed; in Moscow they were just starting up, and talking to like-minded people became a way to test ideas. On the PostgreSQL mailing lists he sent reports and proposals, and one release listed him as a contributor; the talk on how asynchronous logical replication diverges after an incident he and his colleagues first gave in Moscow, then took to a developers' conference in Canada — a format he describes as one big planning meeting, where topics are thrown up on a board. On upstreaming Alexander is level-headed: if a feature is wanted only by you, a fork tailored to your scenarios is cheaper; he then retells a film about eBPF entering the Linux kernel in small increments. From the same thread comes a hackathon devised with a colleague, Pyotr Korobeynikov: participants were handed a broken database, with corrupted bits and indexes, and asked to repair it. The excitement was such that an expert could nearly break a keyboard, and one participant later joined the company.

After six years at Avito, Kostya saw a post saying that people he knew were starting something big and needed expertise in teams and microservices; the pandemic had multiplied delivery orders, and he decided to jump at once. The new company had about thirty people, two or three of them in infrastructure, an unfamiliar stack of Node.js and MongoDB, and an app launch four months out. The data platform came first, and the analytical warehouse was built with help from Kolya, a former manager he plugged into the team for an hour a week — a network moves with you. Then came incident management, where offline pickers and couriers raise the price of any instability, the test-automation and load-testing platforms, and an internal tool grown out of a homemade deployer. When Kostya also ran three product areas for over a year, the teams were reassembled through a collective-farm exercise: every team grows its own cucumbers in a greenhouse, and the leads tell each other what their products, customers and plans are, calling bad things bad. The platform now has about a hundred people and, by his formula, must stay small and take cognitive load off the product teams.