AI Products: First Figure Out Whether You Need AI

Daniel explains his job through the shape of a bank that never had branches: people went to a branch to ask a hard question of a live human, and in an electronics shop an assistant would explain which vacuum cleaner to buy. The world moved to digital, apps and marketplaces appeared, and those people dissolved. His team brings them back two ways: agents that answer any question inside the mobile app, and predictive scenarios where the user need not type at all — if a card is stuck in an ATM, the app meets the customer with a prompt and two buttons. He built his career on a plan made at nineteen: first sharpen general management — stakeholders, entering a new field fast, structuring information — for which consulting was the best place; then learn product next to a strong CEO or CPO who gives feedback daily; and only then work at scale.

The product stage happened at the startup Cooper, called something else back then: about a hundred people in the whole company, founders from investment banking, and a CEO who, by Daniel's account, had worked at McKinsey, done a great deal at Lamoda and graduated from Harvard. He came for that daily feedback and grew there as an IC product manager. Then came Alibaba, and now T-Bank with a team of 250, much of which changed when he arrived: he had to build hiring and create the ML product manager profession from scratch. His main insight about scale is that transitions break everything: he thought thirty people and a hundred were much the same, then that a hundred and 250 were too — wrong both times. Alexander adds from his HighLoad talk on the changing CTO role: at a thousand people domains fall apart, chunks of 250 become autonomous teams needing oversight, and stakeholders contradict each other.

The definition adopted in their AI centre: an AI product is one that stops existing once machine learning is removed. An iPhone or Google camera shoots far better thanks to ML, yet still shoots without it; a chatbot, or the smart camera you point at a person to send them money, does not work at all without the model. Everything is called AI and split by domain: classic prediction such as how much cash an ATM should hold; computer vision, where plain OCR is often enough; and language processing, which changed entirely. A passport photo in the chat can be read by OCR, or for a harder case by a model that understands images. An AI product means all of ordinary product work plus a hundred and one problems from the technology, so the team hires people who have solved them already. Alexander notes that large models have become omnivorous, democratising product building.

Before starting, the team answers three questions. Is AI needed at all — perhaps a heuristic works as well, and the ML product manager, who understands business and technology, must answer that; AI where it is not needed is the standard first mistake. Will the economics work at scale: the demo comes out fine, but unit economics never converge if nobody counted how expensive the cards are. Is the technology ready — by Daniel's account the Oleg assistant was started in 2018, when the idea of a universal assistant was excellent but the best model was GPT-2, which answered a question about the best trainers with something about three kilograms of potatoes, so it was built differently. Success is measured by falling costs — nobody is fired, the company simply stops hiring — by upselling into a huge flow, where the chat tab is among the most visited, and by user experience. A composite metric they call effort counts how many actions a user needs for their job to close, and at the pyramid's base sit evals such as whether it is acceptable to sell anything here.

Why a corporation has it harder: discovery repeats, but at home nobody thinks about scale or economics, and to the executive who built something similar at home two years ago the honest answer is quality at scale. Models still hallucinate and still do not do what is asked; it got noticeably better around half a year ago: there is a metric called instruction following, how closely a model follows instructions you put in its context, and on modern evals the hallucination rate now runs at about four to five per cent, unlike two years ago. The hardest part in a corporation is setting up quality metrics that catch all this. Alexander recalls last year's MIT NANDA study: only five per cent of GenAI pilots succeed, many of the rest losing to homemade automations employees wire together. Daniel answers with an example: support's intent classifier was written for seven, maybe eight years as regular expressions — they cost nothing, precision is high, and with so many of them recall grew too, so beating them with a model on cost proved hard. He also corrects one point: in an application layer you ground the model on the context you pass it rather than let it answer from its weights, or hallucinations multiply.

In a complex domain Daniel calls himself a generalist and is glad when a vertical's lead is stronger than he is. His own work is decisions under uncertainty, helped by breadth of exposure — being wrong is easy, but a twenty per cent hit rate beats two — by pivoting fast while explaining to the team that the information set changed, and by the team's happiness, which drives results. Products live like venture rounds: a gate with a promised value and a date, and at that date either closure or fresh investment. Alexander brings up sunk cost, the suitcase with no handle; Daniel agrees that product managers fear writing a gate that closes a project, although concluding after two months of research that it should not be built is itself valuable. Both live with constant fear of missing out; Daniel copes through purpose, micro-doses of rest and a high upside. Ahead, Daniel expects generative models to take the routine of every role, teams to shrink, the lines between product, analytics and engineering to blur, and middle management to be at risk. In the closing round he names assistants the most overrated trend and automating people's work in specific industries the most underrated, while a moat rests on domain knowledge, audience, viral distribution and speed.