Measuring AI’s Impact on Software Engineering (Category AI)
I watched another episode of The Pragmatic Engineer, hosted by Gergely Orosz. His guest was Laura Tacho, CTO of DX, the developer productivity measurement platform. Just recently, Zhenya Sergeev, Engineering Director at Flo Health, and I discussed the DX Core 4 productivity framework and DX’s new Measuring AI Code Assistants and Agents whitepaper on Research Insights Made Simple. Here are the key ideas from this episode.
1. Criticism of AI media hype Laura argues that most AI headlines are exaggerated and misleading, unsupported by the data from hundreds of companies working with DX. Common problems include:
- Oversimplifying complex engineering processes: engineers do more than write code.
- Confusing acceptance of AI suggestions with code written by AI.
2. The DX AI measurement framework Laura presented a framework I have discussed before. Its three dimensions, ordered by ease of measurement, are:
- Utilization: how actively the tools are used.
- Impact: their actual effect on productivity.
- Cost: the economics of adoption.
3. Actual AI tool usage data DX research across 180+ companies identified these as developers’ most effective uses of AI, in descending order:
- Analysing stack traces
- Refactoring existing code
- Generating code during development
- Generating tests
This contradicts the conventional assumption that AI is most useful for writing new code.
4. Booking.com case study Booking.com reported these results from its large-scale rollout:
- 65% of developers used the tools weekly, above the industry average of 50%.
- Active users merged pull requests 16% more frequently.
- Engineering team productivity rose by 31%.
Investment in training and support was critical to success.
5. The developer satisfaction paradox One surprising DORA finding was that many developers reported spending less time on meaningful tasks. AI speeds up work they enjoy, such as coding, leaving a larger share of time for routine work, meetings and administration. I have a series of DORA posts coming up, with more detail on the research and its findings.
6. Effects on architecture and documentation Teams adopting AI successfully make architectural changes:
- Returning to clean interfaces between services.
- Writing AI-oriented documentation with code examples and no dependence on visuals.
Companies such as Vercel and Clerk create “AI-first” documentation that works for people and AI assistants alike.
7. Structured adoption Highly regulated industries such as finance and insurance show better results through a structured approach to AI adoption.
8. Recommendations for technical leaders Laura advises leaders to:
- Put evidence ahead of hype and headlines.
- Treat AI adoption as an experiment and assess it systematically.
- Invest in training and support.
- Measure developer experience before adoption to establish a baseline.
9. The future of AI-assisted development Laura predicts that roadmaps will give way to more flexible, experimental approaches to adapting to AI.
Laura and Gergely had a very interesting discussion, focused on practice rather than marketing headlines :)
#AI #Software #Engineering #Management #Metrics #ML #Leadership