Measuring AI code assistants and agents (AI column)
I read it fresh. (||yesterday's||) research On the topic of measuring the impact of AI in development from Abi Noda, cofounder and CEO of the DX platform to measure development productivity. This report is interesting, as the guys at DX are among the trendsetters in the world of developer productivity: They sawed down the DevEx model, which I already disassembled.DevEx: What Actually Drives ProductivityandDevEx in Action" The platform team includes Nicole Forsgren, who drove the development of DORA metrics. Accelerate books.I put my hand to it. SPACE framework
Now that it is clear why the guys are interested in studying, let’s look into the report itself and highlight the main ideas.
1. The impact of AI on development AI is fundamentally changing the approach to engineering at technology companies – companies are no longer limited to the number of engineers, but rather the degree to which their capabilities are augmented by AI. These are the results of several companies. Booking.com, implementing AI tools for more than 3500 Engineers have reached 16Percentage increase in throughput for several months Intercom has almost doubled its use of AI code assistants. 41Percentage increase in developer time savings. AI also expands the concept of “developer” – product managers, designers, and business analysts can now create working software with AI, blurring the lines between technical and non-technical roles. Remembering vibe coding:)
2. Key metrics for measurement The authors of the report expand their DX framework with a separate direction of DX AI, where the focus on three things - Utilization Monitor the implementation and use of AI tools. Research shows that even leading organizations only achieve 60Percentage of active use of AI tools - Impact Measure the real impact on productivity. It is recommended to combine: Direct Metrics: Saving Developer Time Indirect metrics: DX Core analysis 4 indicators (PR throughput, percieved rate of delivery, developer experience index) - Cost - tracking costs, net profit (Developer time saved - costs) Interestingly, the guys have benchmarks on these metrics that they provide to DX platform customers.
3. Balance of speed and quality Organizations must balance performance metrics with quality metrics to avoid undermining long-term development speed. AI-generated code may be less intuitive for humans to understand, creating potential problems.
4. Measurement of AI Agents The study recommends seeing autonomous agents as extensions of development teams rather than as independent contributors. Each developer will increasingly work as a “manager” of the AI agent team, and its results will have to be evaluated based on the results of the work of this AI agent team.
5. Approaches to the implementation of the measurement system It is important to properly communicate the goals of implementing AI-related metrics. The authors recommend clearly emphasizing three key points: Metrics will not be used to assess individual employee performance ||(They won't, will they?)|| The purpose of the measurement is to understand how AI-assisted work affects developer experience and software quality, not micromanagement. Data is essential for making investment decisions, helping to determine which tools and workflows bring real value
6. Avoiding traps The report strongly cautions against policy mandates from above or the use of metrics for individual performance assessments. Metrics such as code generation volume are particularly susceptible to manipulation. Proactive communication is key – without it, speculation and fear can fill the information vacuum.
Overall, this report shows the importance of combining AI-specific recommendations with a measurement of overall developer performance to get a full picture of how AI affects organizational performance.
#AI #ML #PlatformEngineering #Software #Architecture #Processes #DevEx #Devops