Data Platforms

Nikolay describes his job simply: all his life he has piled data into one big heap, sat on top of it and handed it out. He began with models — credit scoring, forecasting, "artificial intelligence before it became fashionable" — and kept hitting the same wall: a model is built fast, but wiring it to the organisation's data drags on for years, which is why those projects rarely succeeded. Hence his interest in a data platform carrying reporting, machine learning and anti-fraud. The classic shape is a big project-style warehouse build: goals, tasks, data marts, a team seated to build it; once it is live, data engineers keep loading data and fixing the warehouse while analysts build dashboards. A single-window service where stakeholders file tickets and hear "in a month, in two, next year": complex work stays supervised, but everything queues at one point.

That was the standard back in the previous century, Nikolay explains: data delivery, the database and the BI tool were all paid and very expensive. Yet for a small organisation with a couple of competent people and a dozen analysts he still considers the scheme ideal, though most no longer work that way: at ManyChat, where he works, there are several hundred people and around twenty analysts, with a central data-engineering team and analysts beside it. Some ten years ago the hybrid took over: the platform stays, but analysts spread across product teams and build dashboards inside the sprint, because waiting a couple of months in someone else's queue is incompatible with Scrum. That is how Avito did it; Alexander recounts that at T-Bank the engineers sat in a warehouse structure sliced by streams — credit, debit, payments, marketing — and there were hundreds of them.

The hybrid holds while the central database can take the load: at Avito its ideal form lasted until the analysts numbered around fifty, after which the platform stopped coping with parallel connections. Full decentralisation is Data Mesh: every domain gets its own data engineers, analysts and storage, and may build its own thing or dump everything into a spreadsheet. Nikolay warns that he is giving his own reading and can be argued with: the concept grew out of product thinking and Domain-Driven Design, and a data product maps well onto the familiar data mart. Alexander adds the other two parts — federated governance with standards, so products can be joined up, and a platform layer that removes cognitive load. The author of the concept, as the participants note, was a consultant herself and knew the pure form would not fly; after that comes the phase of "now buy our elephants".

More often the zoo grows by itself: ClickHouse pulling part of the raw data, Spark clusters on the side running their own computations, everything wired to everything. Companies see a mess; a consultant tells them it is a proto Data Mesh, each team owning its databases. Alexander notes the zoo usually grows inside one central team; Nikolay counters that it also grows out of the domains — credit risk on ClickHouse, lending forecasting on Spark — and Alexander replies that the accidental complexity then sits inside product teams. The price is the critical path: an analyst asks what lag his number carries and hears "two days", because the Spark data product only runs after the products in Greenplum and PostgreSQL, while an error in someone else's mart is fixed "in three months, write us letters". The European data-engineering crowd jokes that Data Mesh turns into Data Mash: the metric was written by someone who has since left, before him by another, and nobody wants to dig in. The cure is a base domain holding clients, money, employees and units, so the path shrinks to two or three steps.

A separate thread is what happens to the databases themselves. Startups compute analytics straight on a microservice's production database and do not worry; what moves that from dangerous to acceptable is the analytical replica — it is idle anyway, its processors are free, and the analytics runs there. Nikolay took this to X5 Digital when they hit the replica's limits; Alexander had heard about the visit from Kostya, but not about the concept. The end used to come at analytical tables of 10–20 million rows, now the threshold is nearer a hundred million; PostgreSQL and Oracle are single-master, replicas take reads but not writes. In come MPP databases: a database from outside, a ring of servers inside — ClickHouse, and Vertica, whose development has by the latest news been stopped. Their ceiling, though, is concurrency rather than volume: because of central-catalogue synchronisation a "select … limit 10" costs no less than a heavy windowed query, and inserts top out at around ten a second against thousands to tens of thousands in PostgreSQL. There is one cure, a full copy of the cluster for millions of dollars: Avito made one at 50–60 analysts.

Then comes the next turn of the wheel: splitting compute from storage. Alexander recalls an article explaining it by the network outpacing disks in speed; Nikolay thinks it was mostly a fight against fear. The first solution, Athena, is crude: for a query against data in S3 a cluster is raised from scratch, pulls the data, returns the result and dies, and you wait three hours — and analysts turned out fine with that, since they do not need millisecond latency. Nikolay does the arithmetic aloud: a billion rows and a hundred columns in Parquet compress fivefold, five of the hundred columns are actually needed, some ten gigabytes remain, and a 100-gigabit network moves that almost instantly. Hence the picture: S3 as storage, with ClickHouse, Trino and other engines on top, tables picked up from Iceberg, layers arranged as a medallion architecture. ManyChat runs Snowflake everywhere for analytics; in companies on Databricks each domain has its own Spark cluster and analysts are not allowed to write queries themselves, because one creative query eats everyone's resources. The future, for Nikolay, is personal compute modules — while the world has turned twice over: Hadoop with Hive promised the same, only now S3 stands in for HDFS and SQL is back on vectorised engines.