Does AI Actually Boost Developer Productivity? (100k Devs Study) (AI column)
I saw something interesting. 20 minute speech Yegor Denisov-Blanch of Stanford University, on measuring the productivity of engineers. Egor presented the results of a three-year study by Stanford on the performance of 100k developers 600 companies. The results of the study are as follows: AI increases productivity on average 15-20%, and the effect varies dramatically depending on the context of the task, the maturity of the code, the popularity of the language and the size of the repository. Below are the main thoughts of this talk + study-page
1. Why are current productivity estimates unreliable? Most published works are funded by tool suppliers (GitHub Copilot, Sourcegraph Cody)This distorts the sample and metrics. Focusing on the commit counter/PR without taking into account the size and quality of the tasks leads to a false growth of output. (We have a lot more code, but how useful is it?) Greenfield and toy projects overstate AI’s benefits: LLM will easily generate boilerplate, but real enterprise code rarely starts from scratch Productivity surveys have problems—self-reported productivity is rarely accurate, although other factors such as satisfaction or well-being surveys can be collected well.
2. What did the Stanford methodology look like? Connecting git repositories, among which were mainly private repos, this allowed to close the context of the team Evaluation of the code by a group of experts, where 10-15 Architects give points for quality, support, complexity (By the way, their scores correlated well with each other.) In fact, this is the creation of markup for further supervised learning. Training of the model for reproduction of expert assessment with high correlation for the sample of projects evaluated by experts - this allows further scaling of this model without expensive review Classification of changes - the authors wanted to classify changes into groups "addition of functionality", "deletion", "refactoring", "rework" (reworking) This model is applied retrospectively with 2019 along 2025 a year to track the impact of COVID, LLM implementation, etc.
3. The main numerical conclusions of the study Raw code growth after AI+ implementation30-40% - this includes useful and "rework" volume Improved productivity gains +15-20% is corrected for bug fixes On complex tasks, the increase is slightly higher. 0% with high variance (sometimes) If you think of it as a 2x2 matrix, ||how consultants like||We get the complexity of the task × maturity of the project.
Complexity/Type. Greenfield Brownfield Low-complexity 30-40percentage 15-20percentage High-complexity 10-15percentage 0-10Percentage of benefits, sometimes loss Efficiency is influenced by language – for popular languages, LLMs work better, and for esoteric languages they work worse. Efficiency is affected by the size of the codebase, and the benefit falls logarithmically with increasing volume. The hypothesis is that this is influenced by the limitations of the LLM context window, the growth of noise and a large number of dependencies within the code. (coupling)
If you try to give practical advice, it seems like this. Assess the typical complexity and maturity of your project before the large-scale implementation of LLM assistants Try to use popular technologies - the best models are trained on them and the quality of tips is higher For legacy monoliths, pilot on small modules - productivity may not increase with the help of assistants out of the box (ala basic Cursor) Monitor "rework"-share: a sharp increase in the amount of code can create the illusion of productivity Combine a quantitative git analysis with a qualitative review, rather than referring only to a “team satisfaction survey.”
P.S. Yegor Denisov-Blanch is the author Research on "ghost" developersThat's what I heard last year. However, I could not find the study itself, but there are posts about it in a large number of places.
P.P.S. Compare it to design. experimentfrom METR:)
#Engineering #AI #Metrics #Software #DevEx #Productivity