MLOps in Theory and Practice

Alexander opens with an honest framing: students are shown models and architectures on clean data, where it is clear in advance which model fits and who will give feedback, while in real life a huge underwater part of the iceberg sits beneath and decides everything. As a scaffold they take a Google Cloud article almost three years old, full of good illustrations. They start with history: some fifteen years ago a fence stood between development, which wanted to ship value faster, and operations, which wanted reliability, and it was torn down in the name of time to market. Dima recalls that in his experience the admins were simply abolished and developers picked up the whole routine full-stack, which flew faster and better. Zhenya adds that in ML the gap was even sharper: the people building models sat in their own pen with their own codebase, handed the result to developers, and rollout was done by yet a third group on site, with whom the link was completely broken.

The second wave is data. The participants walk through DataOps: the warehouse, speed, reliability, data as the new oil, and data scientists who used to sit in a pen of their own. Zhenya notes universities have long taught warehouse building, but he has never seen a systematic course on data quality, and companies rarely solve it either. One of the participants jokes that wherever a decent DataOps has not taken root, MLOps usually smells too, and not of the right thing. Dima describes the move from a classic warehouse to ELT and a data lake, and then to Data Mesh: the concept, as he recalls, appeared five years ago or already in 2018, and it does for data roughly what microservices and domain-driven design did for applications. Ownership of raw data quality moves into product teams, the data-delivery contract is checked in their own pipeline, and incidents are fixed by the authors rather than by firefighters hunting for the burst pipe. In the bank, Dima says, that transition is under way: the tooling exists, some people have moved, and the main push was planned for this year.

Next comes the ML lifecycle diagram, and Alexander asks the specialists whether it looks plausible. The answer: rename the sectors and you get roughly the same loop as for an ordinary application — experiments, collection, integration, evolution — except that what changes is data rather than lines of code, and monitoring maps one to one. Only one branch is genuinely specific: incrementally developing a model on richer feature sets resembles building new functionality, whereas regular retraining so the model does not degrade under a changed environment really is different. Zhenya argues the diagram lacks a step for rethinking the goal: requirements for models change more often than for well-understood services — quality gets measured differently, people want something else from the model, the data changes completely. In software the criteria are usually formalised as milestones, while with a model you can build something and only then learn you were heading the wrong way; uncertainty and variability are higher here.

Dima goes through the platform's capabilities stage by stage. Prototype: spinning up a machine and installing libraries should have a minimal barrier to entry rather than require a tambourine. Then real data: computing features and targets and reshaping them for training, both of which run into compute and data access. After offline validation the model either rides the standard rails of an inference service that scales horizontally by itself, or needs custom work when business rules are wrapped around it. Online validation on real users, Dima says, is often skipped and that ends badly: an experimentation platform is a mandatory prerequisite, with its own science of slicing users and getting a trustworthy result. The loop closes with support: scheduled retraining, versioning, and a record of when the model was updated and how its quality moved. Dima has never seen a case where all of this comes out of the box. Zhenya notes there is marketing in the picture, and Alexander agrees: such documents are written so that you settle onto the vendor's tooling.

The most ML-specific piece, in Dima's words, is retraining. His example: an anti-fraud model was rolled out and started banning people, and those banned switched to methods the model had never seen in training, so quality drifted down. Degradation also comes from your own bugs: a feature such as click-through rate over a time window slowly converges to zero because impressions are written and clicks are not. Hence the requirement to monitor both the model's output and its input and to keep a system of trigger events: either retrain automatically or raise a red flag and go investigate, because unlike a hard interpretable rule a model cannot be inspected by eye. Alexander brings a fresh case: in a one-to-one with the analytics director Kirill Nikolaev they discussed a model working on the flow left over after manual empirical rules. They wanted to remove some rules and widen the flow, but the model was tuned to the old one and its quality would drop badly on the wider stream. One of the participants qualifies it: a quantum leap is needed here — strip the rules off part of the flow, collect data, retrain.

Zhenya then walks the same blocks from the angle of what a university can actually teach. Data preparation: poorly. There is a decent course on crowdsourcing, but the ability to look at data and see patterns he considers closer to an art. Training and evaluating models are taught excellently, at any level, with or without mathematics. Serving is harder: no settled framework has emerged, there is no PyTorch of this area. Online experiment theory is covered well, though in his impression no open A/B testing platform exists. Monitoring is worth a lecture or two, and their first systems design course will only launch in the next academic year. Alexander proposes labs where a student wraps a model in metrics while the teacher feeds in different traffic and waits to see whether the problem is found; he recalls his own Kubernetes administrator exam — as he remembers it, 24 tasks in three hours, then 18 in two, with a resit every two or three years. Both sides end on the same note: breadth matters, no engineer should stay isolated, and Alexander and Zhenya agreed to record a separate episode on exactly that.