AI Impact: 90% per Step, 5% per Cycle

The episode opens the podcast's second season with introductions: the guest is a director of adoption at a company building its own assistant and agents for developers, and before that spent several years as CTO inside a consultancy, where about three years ago he was told to work out what AI was good for. All last year he toured conferences with a talk about the tools at each stage of the SDLC, updating it monthly because new products shipped faster than the slides went stale. Alexander answers with a parallel story: a year ago he was popularising AI-assisted development and owned one of the streams around architecture, and this year he moved into that role full time, running adoption at T-Bank. He says the sceptics disappeared in the spring; the guest corrects him — more like late autumn, once people had tried the tools and the claim that this does not work stopped being tenable.

The guest describes the typical failure almost as a joke: two enthusiastic engineers go and argue to management that AI is needed, get asked how they will measure the business effect, are handed a gaming PC and told to build a production system. Out comes a contraption users hold to a higher standard than any industrial product: Claude Code can do this, and the thing you knocked together in three months somehow cannot. The systematic route starts with a pyramid of metrics: a CTO with P&L and ROI at the top, engineers with DORA and unit tests at the bottom, managers who want velocity and shift left in the middle — all looking at the same thing with different eyes. The first layer is plain telemetry: how many people use the tool and how many requests they send, which gives adoption. The second is the request and response logs, stitched into units of work, dumped into ClickHouse and pulled into dashboards and Grafana.

Each unit of work is then priced in time and money at an average rate, so a team sees what a month would have cost had everything been written by hand: six million saved, ten per cent of efficiency. Alexander calls this counterfactual analysis and asks where the estimate of what did not happen comes from. It grew out of an earlier project of the guest's: an agent that broke an analyst's task into subtasks, went into the code and produced its own estimate. The plan was for people to correct it and tune coefficients; in practice it was accurate from the start, more accurate than T-shirt sizes and story points at groomings, because it arrives with a breakdown and a justification. A well-known code-volume formula, with coefficients for project complexity and team qualification, proved less relevant: an estimate from a coding-tuned model matches team expectations in roughly 80% of cases, and any metric can be drilled into to see who was doing what at three in the morning.

The same logs produce an unexpected metric — an emotional one. The system scores how the user writes: neutral, aggressive; a weekly chart shows a spike of negativity, and inside it sits a bug-ridden part of the project or a problem with the analysis. The guest notes that nobody asks an engineer how well they slept when estimating a task, whereas here their state is visible in the logs. The logs also show that writing code is far from all of the work: at first people explore the codebase, documentation and the tracker, and ask about business terminology, such as what derivatives are and how they are handled. When ten people struggle with the same thing, it can be closed in one move. Then a trust scale kicks in: people start with work that does not touch production, so statistics skew towards research, and once trust appears that share collapses to ten or twenty per cent while coding and testing take over. Why it collapses, the guest admits he does not fully understand.

The adoption programme rests on nine metrics watched daily by the masters who coach the teams. The first is engagement: if nine of ten people in a pilot sent three hundred requests each yesterday, that is a good sign, while whoever dropped out is worth a conversation. Effectiveness is counted across six categories — coding, testing, review, documentation, onboarding and one more the guest forgets on air — as the share of AI time that produced value. That share is then recalculated onto the whole SDLC: a task done right the first time saves some ninety per cent at that step, but across the full cycle it comes out at about five per cent, or even one. The metrics themselves he calls monitors rather than insights, so in the third week the dashboards are read by hand: that is how it emerges that some teams need a planning mode, others need shared microservice projects pulled into context, and somewhere the security team has closed part of the project off.

Counter-metrics get their own turn. Code acceptance is measured too, brackets and blank lines thrown out; across the industry it sits around 30%, though a very high share is not always good — sometimes it means people generate without looking and skip review. This is not an autonomous agent in a pipeline but a chat next to the code; they removed autocomplete altogether, and tap-tap-tap, in the guest's words, is marketing: a programmer is busy with architecture rather than completions. Alexander adds figures from Meta engineers shown at a developer productivity summit in late 2025: while up to 30% of the code is generated, diff authoring time does not shrink, and the meaningful tens of per cent appear closer to 60%, because the cognitive cost of making sense of machine-written code eats the benefit. He himself spent years owning developer productivity in a company with more than ten thousand engineers, and reminds that beyond DORA there are SPACE and DevEx. Then come the roles: the champions now ask which to keep when eighty per cent of time goes into coding.