Early-2025 AI and Experienced Developer Productivity

The host opens with the frame: METR, an organisation that benchmarks models, set up an experiment at the start of 2025 and measured the effectiveness of engineers who modify open-source frameworks. The result was a statistically significant slowdown, the report spread across the industry and people began citing it. Alexander ignored it at first, then did write about it on his channel and on Medium, but he wanted to discuss the paper with someone who understands how such experiments are set up and how their validity and statistical power are judged. Artem introduces himself: for the last few years he has worked on development efficiency and Developer Experience — first at Sber, now at Yandex, where he is effectively the CPO of the developer platform in the city services. He started on AI in development back at Sber, actively in 2023, and stresses it is not his only topic but one tool for raising efficiency.

Alexander read not only the digest of results but all fifty pages — down to the advertisement used to recruit people. They looked for experienced contributors to large open-source repositories, paid them a substantial hourly sum, asked them to record their screen and to take real tasks from those same repositories, so that the work passed code review and was merged into the main branch. In total 246 tasks and 16 engineers, randomly split into those allowed an assistant and those not; before a task a person records an estimate, afterwards the actual time. The activity labelling, by Alexander's account, was probably done by the researchers themselves, who watched many hours of video. Tasks, as he remembers it, ran to roughly one to four hours, and large ones were decomposed. There was definitely an exit interview; on the way in, rather a screening. The experiment cost, he does not remember exactly, either one or two hundred thousand dollars.

The first thing Artem thinks about is who sponsored the study. Its focus, as far as he remembers, is on Anthropic models, and he describes the company's logic like this: first come the frightening studies about what a model is prepared to do to survive, and then the benchmarks show that the most humane and safest models are Anthropic's own — hence his question about how much bias there is. Alexander pushes back: it looks like they simply took the most popular tool, Cursor, and the best coding model of the moment, Claude 3.5 to 3.7, and wanted to measure by hand rather than by surveys or from inside a company where the CEO talks about unicorns. What they tested was assistant mode, where the developer writes the code and the IDE helps. Artem answers as a manager: given a KPI to prove AI is ineffective, he would take sixteen people (or nineteen, he no longer remembers), a deliberately large codebase and experienced developers who know it.

Alexander extends the fantasy: he would also buy the tool for anyone who had never used an assistant and tell them they are being made to use it. Artem qualifies that it need not have been so — he is reading external signals; were he this team's manager, he would have asked for control groups across different tasks and processes. Then comes the estimate gap the headlines grew from: by Alexander's retelling, those forbidden the assistant allowed about two hours per task and beat that, while those allowed it expected an hour and a half and on average spent more than two; the confidence interval does not reach zero. Artem sees the usual bias here — nobody can estimate tasks, we are all optimists — and would like to know whether there was an entry survey about expectations, since a believer lowers the estimate and a sceptic raises it. The paper is full of caveats, down to a table of what the authors do not claim to prove; expectations were different — economists assumed a 40% speedup. But caveats go unread: journalists take the abstract and produce the news that AI slows developers down.

Artem admits he did not read the fifty pages: he dropped them into ChatGPT and talked it through. The video labelling let the authors speculate about causes: some eighteen factors were collected, of which around five are treated as the most significant. They are over-optimism; the engineer's familiarity with a repository of over a million lines that does not fit into context; the size and complexity of that repository; low trust in suggestions among people who had never used such tools; and the project's implicit rules — open source rarely writes them out for an assistant, an engineer absorbs them through rounds of pull-request feedback, and now they have to be made explicit in MD files. Alexander compares the setup to a match between basketball and football players played by one sport's rules, and Artem recalls the time spent cleaning out changes that did not apply: by his recollection, less than 40% of the generated code was used.

The people who benefit most from reading this, Artem argues, are managers: any corporation carries a codebase written over years under the primacy of deadlines, and where software is not the core business there is even more legacy — an assistant pointed at it will not give the promised boost. His long-standing thesis: projects fail on integration and communication, because we convey context badly even to each other. Alexander answers with practice from his write-up of an interview with Boris Cherny of Anthropic, the creator of Claude Code: trust is built as a ladder — first ask it to describe the codebase and check that against what you know, then simple bugs, then medium tasks through a reviewed plan, and only then hard ones, where the agent builds a prototype while you drive the work. Artem says this is literally onboarding a new developer. The host's verdict: a decent benchmark that lacks scale, and the methodology is more interesting rolled out inside large companies.