[2/2] Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (AI column)
Continue. posture About the study, I will tell about the results obtained Statistically significant deceleration of the 19% when using AI tools with 95% confidence interval The authors separately pointed out the limitations of the generalization of these findings.
- Developer experience. Results specific to experienced developers (5+ years of experience with repositories). For less experienced developers, the results may be the opposite. Size and complexity of codebases. The study was conducted on large, mature projects. (1M+ lines of code). On smaller or new projects, AI can show a positive effect.
- Acquaintance with the project. The developers worked in familiar repositories.
- Task type. Tasks have already been decomposed to a size of no more than 2 hours, which may not reflect the full range of development tasks.
- External validity. The results do not mean that AI tools are useless in all development contexts.
As a result, the authors drew the following conclusions:
- AI tools slow down experienced developers 19% when working in familiar codebases, which is contrary to expectations as developers (predicted the acceleration of 24%)and experts (predicted the acceleration of 38-39%).
- They also thought about slowing down factors. 5 The main ones, although they made such a remark.
However, we strongly caution against over-indexing on the basis of any individual pieces of evidence, as we are not powered for statistically significant multiple comparisons when subsetting our data. This analysis is intended to provide speculative, suggestive evidence about the mechanisms behind slowdown. These are the factors. Excessive optimism about the usefulness of AI High familiarity of developers with repositories Large and complex codebases Low reliability of AI (accepted44percentage) Implicit context of repositories not available to AI
As a result, the authors emphasize that the results do not mean that AI tools are useless.
Now let’s talk about the problems of the study and why its results should be taken with caution. 1. Small sample size Only 16 This limits the statistical power and calls into question the representativeness of the sample relative to the population. The setup of the experiment did not allow to answer the questions which factors influenced the results. 2. Short-term The study does not take into account the long-term effects of learning to use AI tools. 3. Specificity of context Large open source repositories were selected, suggesting that the results may not generalize to other types of projects by size or specificity. (web, mobile development) 4. The Hawthorne Effect Participants were aware that they were participating in the study, which may have influenced their behavior. 5. Subjectivity of measurements Time to complete tasks was measured by the developers themselves, which could make systematic errors. 6. Determination of productivity The study focused only on runtime, without considering other aspects of productivity: code quality. (The main thing is to go through code review.)job satisfaction
Overall, I think the experiment itself is interesting, but I'm more of a believer in measuring practical effects in organizations where there's a measurement of developer productivity and AI there is added to the developer ecosystem gradually and through a/b experiments on a large scale that allow for tracking effects. Specifically, such approaches can be read in posts. About Google and their approaches in a series of articles "Developer Productivity for Humans" (More in the posts: 1 and 2) About the approach of the banned in Russia company Meta, which they described in the articleWhat's DAT? Three Case Studies of Measuring Software Development Productivity at Meta With Diff Authoring Time" (More in the posts: 1, 2 and 3)
- Well, in case of an emergency, you can watch my performance.Why Do Developer Productivity in a Large Company?"
#AI #ML #Management #Leadership #Software #SoftwareDevelopment #Architecture #Metrics #Devops #Processes