Can you prove AI ROI in Software Engineering? (AI column)
Just recently Egor Denisov-Blanche (Yegor Denisov-Blanch) spoke AI Engineer Code Summit 2025 In New York with this provocative report:) In fact, this is a teaser of the results of a two-year study of the impact of AI tools on the productivity of developers, covering 120 000+ developers in 600+ companies (||At the very least, this coverage inspires confidence.||). Previous Previous post: Does AI Actually Boost Developer Productivity? handler And some of the theses are completely repeated, or rather questions of the type 1. Why are current productivity estimates unreliable? 2. What did the Stanford methodology look like? 3. The main numerical conclusions of the study
But there are new results in this report.
**1. The median increase in productivity is10%**But the gap between leaders and laggards is growing. 2. Quality of use of AI is more important than volume. Correlation between the number of tokens spent and productivity growth weak (R² = 0.20). Moreover, there is an effect of the "valley of death" at the level of 10 Millions of tokens per engineer per month – teams using this amount perform worse than those using less. 3. Clean code base is critical for AI. Environmental cleanliness index (Environment Cleanliness Index), taking into account tests, types, documentation, modularity and code quality, shows the correlation R2 = 0.40 Increased productivity from AI. Pure code enhances the effect of AI, and technical debt neutralizes it. 4. AI accelerates codebase entropy. Uncontrolled use of AI accelerates the accumulation of technical debt, which shifts the code base into an area where AI is less efficient. Active code quality management is required to maintain the benefits. 5. Access to AI и Effectively Usede. In a case with two business units of the same company with equal access to tools, the use was very different. It is important to measure not only the fact of use, but also how engineers apply AI.
The author proposes a methodology for measuring ROI in two parts. 1. Measurement of use Access-based (gainer) vs usage-based (API telemetry). Usage-based is the gold standard that allows retrospective analysis through git history. 2. Measurement of engineering results Primary metric: Engineering Output - ML model that replicates panel evaluation from 10-15 Independent experts on complexity, implementation time and support Guardrail metrics: Rework & Refactoring, Code Quality & Risk, People & DevOps – metrics that need to be kept healthy but not maximized
Ironically, the author strongly opposes the use of PR counts, lines of code and even DORA as the main metrics of productivity.
Well, if you analyze the author’s approach, it contains strengths and weaknesses. (+) Research methodology looks solid. An ML model that replicates expert assessments tested for correlation with real panel data is the right approach to measuring complex qualitative aspects of code. (+) Understanding System Effects. The codebase entropy management concept, the link between code purity and AI efficiency, understanding what engineers need to know when not to use AI (+) Criticism of DORA as primary metric is justified. DORAs are process indicators, not results. Maximizing them can be harmful. (Remember Goodhart's Law?) (+)AI Practices Benchmark with maturity levels (From personal use to agent orchestration) It shows an understanding of the evolution of practices. (-) The author's background is not a pure development He has no experience or education as an engineer. (-) ML Model as a Black Box. Although it is claimed that the model correlates with expert assessments, details of methodology, dataset for training, model quality metrics are not disclosed in the report. (-) Environment Cleanliness Index Experimental. How exactly are the components of the index weighted? How is it validated other than the correlation with productivity?
#Engineering #AI #Metrics #Software #DevEx #Productivity