Research Insights Made Simple 17 - Measuring the Impact of Early-2025 AI on Developer Productivity (Filed under DevEx)

Research Insights Made Simple #17 - Measuring the Impact of Early-2025 AI on Developer Productivity (Rubric #DevEx)

Analysis of the report METR "Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity, which shows developers slowing down when using AI. The methodology seemed interesting to me, but the sample size is already in 16 I thought engineers were too small to make loud statements about slowing down development. However, this sample size did not prevent many journalists from actively writing about this study. As a result, I invited Artem Aryutkin, a SRO platform for developers in Yandex City Services, with whom we discussed all the pros and cons of this study.

Here is a list of what we have discussed. 40+ minutes Introduction, announcement of METR and benchmark

• Meet the guest. Research sponsor and possible bias
• How the experiment works. Hypotheses about research motives and design Evaluation and self-assessment of tasks by participants Recruitment and requirements for participants
• Scope and methodology limitations
• Figures: 246 task 1 before 8 hours in duration, 16 researcher
• Five factors of slowdown Context, integration and communication as a bottleneck How to work with the tool by difficulty levels Why Models Are Difficult to Apply Changes to Real Code Benchmark results and limited generalization

The podcast release is available in Youtube, VK Video, Podster.fm, Ya Music.

P.S. Nana Big Tech Night next Friday, Stanislav Moiseev, my colleague who runs the RnD center, tell It's about different approaches to measuring developer productivity in general, and how to measure the impact of Gen AI on that. If you are interested in the topic, register And come and listen.

#Software #Engineering #Metrics #Databases #Architecture #Devops