Measuring the impact of AI on software engineering (AI column)
I saw it. podcast "The Pragmatic Engineer" by Gergeli Orosz (Gergely Orosz). Laura Taco came to visit him. (Laura Tacho), technical director Productivity measurement platform for DX engineers. Interestingly, just recently, Zhenya Sergeev, an engineering director at Flo Health, and I discussed Research Insights Made Simple. DX Core 4 framework for measuring productivityAnd new. Whitepaper "Measuring AI Code Assistants and Agents" from DX. But if you go back to this podcast and its key ideas, here they are.
1. Criticism of media hype around AI Laura emphasizes that most media headlines about AI are exaggerated and misleading. They are not supported by actual data from the hundreds of companies DX works with. The main problems with such news are Simplifying complex engineering processes beyond recognition (Engineers don’t just write code.) Mixing “AI Adopts” with “AI Written Code”
2. DX AI framework for measuring the impact of AI Laura introduced a framework to measure the impact of AI (I already am. handler). It has three types of parameters, which are listed in order of simplicity of measurement.
- Recycling. (Utilization) How actively the tools are used Impact. (Impact) What is the real impact on productivity
- Cost. (Cost) Economic efficiency of implementation
3. Real data on the use of AI tools DX-based study 180Companies have identified the most effective scenarios for the use of AI by developers (descendingly)
- Stack trace analysis Refactoring existing code
- Code generation in the development process
- Test generation This result contradicts the conventional wisdom that AI is primarily useful for generating new code.
4. Case study: Booking.com Booking.com conducted a large-scale implementation of AI tools with the following results:
- 65Percentage of developers use tools weekly (above industry average 50%)
- 16% increase in the frequency of pool request mergers among active users
- 31% increase in productivity of engineering teams Investing in training and support is critical to success
5. Developer satisfaction paradox One of the most surprising findings of the DORA study is that many developers report reduced time for meaningful tasks. Reason: AI speeds up parts of work that developers like (code-writing)Leaving more time for routine, meetings and administrative tasks. By the way, I will have a whole series of posts about DORA, where I will tell you more about the research and insights from them.
6. Impact on architecture and documentation Teams that successfully implement AI make architectural changes: Return to clean interfaces between services Creating documentation focused on AI (with code examples, without visual dependencies) Companies like Vercel and Clerk are building AI-first documentation that works for both humans and AI assistants.
7. Structured approaches to implementation Highly regulated industries (finances) They show better results thanks to a structured approach to AI implementation.
8. Recommendations for Technical Leaders Laura gives the following advice to managers: Data is more important than hype – based on facts, not media headlines Consider using AI as an experiment – evaluate effectiveness systematically Invest in training and support – critical to successful implementation Measure developer experience before implementing AI – this creates a baseline for comparison
9. The Future of Development with AI Laura predicts that roadmaps (roadmaps) As teams move to more flexible, experimental approaches to adapt AI
In sum, Laura and Gergeli had a very interesting conversation about the impact of AI, focusing on practice rather than marketing headlines:)
#AI #Software #Engineering #Management #Metrics #ML #Leadership