[1/2] How tech companies measure the impact of AI on software development (AI column)
Interesting. posture Gergeli Orosz, a subscription author for The Pragmatic Engineer, and Laura Tahoe, a CTO of the DX platform that measures productivity. They were on the podcast together.Measuring the impact of AI on software engineering"which I already am." handler. DX has its own. framework A framework for measuring productivity and recently appeared to measure the impact of AI. In the Research Insights podcast, we discussed them with Zhenya Sergeev, engineering director at Flo Health.
If you go back to this article, then interesting examples of specific metrics from 18 The companies Laura cited as examples. They show that AI tools have already spread widely: they are used.85% of developers. But measuring their effects is not easy. 60The percentage of techlides admit that they do not have clear metrics of AI benefits. The news simplifies statements like “AI writes N% code.” However, lines of code are a poor productivity metric: source code is more of a burden than a value.
As a result, Laura suggests focusing on the basic metrics of productivity: quality, reliability, speed, satisfaction of engineers. Many companies continue to measure the same metrics as before AI (Failure rate, release rate, cycle time, DevEx)They also assess how much AI helps them improve. If these basic metrics are not defined and there is no consensus on what is considered productivity, then the AI score will be reduced to empty metrics such as “the amount of code” or the percentage of clues accepted.
In addition to standard indicators, AI-specific metrics appear and they are divided into three categories. (details of the article "Measuring AI Code Assistants and Agents"):
- Level of use (usage)
- Impact on work (hours saved)
- Economic impact (cost & benefits)
Continuation in the next post.
#AI #ML #PlatformEngineering #Software #Architecture #Processes #DevEx #Devops