Data Platform: Changing Planes Mid-Flight

Sergey introduces himself briefly: he owns all technical development of the bank's centralised data platform. Then the conversation goes back: born near Kharkiv, raised in Gatchina, the last two school years in a border-service cadet corps, after which he skipped the military track and studied programming at an electrotechnical university in St Petersburg. His father pushed him towards surgery, but Sergey found that too much responsibility and chose a trade where the result of your work is visible to the naked eye. He started as an intern around 2006: hired for C++, but the vacancy closed and the director sent him off to write .NET instead. He stayed eight years, first owning the reporting system and then joining the application optimisation team, where the bottleneck almost always sat in the data layer. A database of 800,000 tables and some two terabytes felt enormous at the time. From intern he grew into a team lead of seven or eight people.

He joined OK for the volume: to see how a service lives when every click, login and video view is logged. The task was to retire more than two hundred MS SQL servers that computed all the statistics with a five-minute lag: Microsoft had started asking far too much money, and single tables no longer fitted even a server with the freshest hardware. The team was called DWH, about ten generalists: on-call, buying hardware, the ETL framework, dashboards, code in Java and .NET. A couple of years later people with that breadth stopped being findable, so the team split into three — the infrastructure one, which came to be called the data platform, the DWH team and the analysts. Sergey took the platform: over a hundred petabytes in HDFS across seven data centres, petabyte-scale Kafka and ClickHouse, all held by up to eight people. Everyone owned one direction for long stretches, everyone was on call for everything, data-centre failure drills were routine, and there was no time for pretty interfaces.

The storage was closest to a data lake: almost no metadata, no Hive metastore, daily logs turned into aggregates, and elaborate layers such as Data Vault or anchor modelling were unaffordable at those volumes. The graph ran about two thousand calculations a day, with over a hundred experiments and hundreds of metrics. The interface of the A/B platform was Jira: you open a ticket, describe the split and the methodology, press the button, and the report arrives as a comment on the same ticket; details were looked up in Grafana. Sergey offers a thesis: a company that goes hard on experiments is probably not growing that fast — when everything is green, you would have seen it on the charts anyway. The interesting part starts when half the metrics are red and somebody has to call the trade-off acceptable: loyalty against money right now. One story stands apart — the red link to the gift sale: leave the colour alone and it becomes white noise, change it often and it reads as advertising, while a rare abrupt change boosts revenue hard.

There was no leap at OK — volumes, the hardware fleet and quality grew, but no new concept arrived. Sergey was then made a director and given not only the OK platform but also the VK platform and the central data office. He describes VK as heavily federated, with businesses that barely intersect and do not want to know about each other, which gets in the way of data. About the appointment he is unromantic: positions opened up, people left, things were falling apart, and his own platform kept working. The job turned out to be firefighting — the OK team had turned over completely and had to be replaced. He left when his direct manager changed and decisions started coming from above without his technical input; he spent a month and a half collecting offers and picked T-Bank. On adoption: OK had more data, but three to four hundred people used the tools, while at the bank the count runs to 15,000 and 12,000 users of two systems.

Sergey puts numbers on the bank's platform: around 230 developers, some 152 petabytes across the storages and roughly 100,000 compute cores. What won him over was the centralisation — after VK it looked like a treat — plus a clearly defined change ahead: the architecture stood on Greenplum, which the team knew would not scale further. He explains the design: a classic MPP, equal hosts, segments on each, a segment being essentially one PostgreSQL; a query is scattered through the master to all segments at once, so it runs at the speed of the slowest node, and data has to be spread as evenly as possible. The more hosts, the higher the chance of a failure: the ceiling is 72 large nodes, plus a limit of roughly a hundred parallel sessions, so clusters were grown by budding — about ten now, from 30 to 72 nodes, with a data bus and a standby. Alexander translates this into architecture terms: the blast radius stretches across the whole cluster, and the standby is a legacy of the corporate world, since under the same load it goes down next.

The new platform separates compute from storage: data in S3 from in-house colleagues, Spark and Trino as engines in Kubernetes run by a neighbouring team, Parquet with Iceberg on top so users keep working with tables. Sergey likens it to a plane: the old one is in the air with every business process running, while a new one is built alongside, and people have to be moved across without anybody falling to the ground. The hard part is not building it but moving 20,000 sources and thousands of regulated processes without breaking the reporting. Ad-hoc analysts barely notice the switch — they patch some syntax and ask why the new system is slower; the answer is that it is more scalable and more predictable. The second change is data contracts across all deliveries, in their second year: the supplier changes the contract first, the data afterwards. Data engineers were handed to the domains last year, while hardware stayed with the platform — although an honest platform, Sergey says, should hand out building blocks like an internal cloud.