[2/2] How tech companies measure the impact of AI on software development (AI column)

Continue. posture About measuring the impact of AI on development, I want to share an example from Dropbox where90Percent of engineers regularly use AI assistants (by industry ~50%). It measures active users, CSAT, time savings and AI costs, and then overlays that data on core metrics: change fail percentage (CFR), throughput et al. (||CFR and throughput are counterbalancing metrics.||). As a result, AI users deliver to ~20% more code per week, while the quality does not fall (downtime). It’s a sign that mass adoption of AI is actually increasing efficiency, not just activity.

Companies also compare the performance of those who use AI and those who do not, and observe changes over time. Such an analysis is only possible if basic data are available before AI is implemented, but it is valuable: it can test hypotheses about the impact of AI on real numbers. It is important not to focus only on speed, scoring on quality - it is important that the increase in the frequency of releases / PR is not accompanied by an increase in bugs or rollbacks. To identify the hidden negative effect (Deterioration of maintainability, dissatisfaction of the team) Conduct regular Developer Experience surveys.

From the reflections of the authors follow practical recommendations:

1. Define Key Productivity Metrics (quality and speed) Set a baseline before implementing AI. Without it, it’s unclear whether AI is improving what it needs.
2. Keep track of AI usage, but don’t confuse activity with outcome. What is important is not the amount of code generated, but the impact on the quality, speed and comfort of work.
3. Balance speed and reliability. Acceleration of work should not lead to the growth of marriage - watch for speed and quality metrics.
4. Consider the team's experience. Along with logs and other system metrics, regularly collect developer feedback. (DevEx polls, CSAT) So you don’t miss problems that are invisible in numbers alone.
5. Don't punish metrics. Demonstrate that AI metrics are for improving processes, not evaluating personnel. Otherwise, you will undermine trust and get distorted data.
6. Cultivate experiments. Try different tools on small projects, compare where the effect is maximum. Encourage knowledge sharing – discuss successful and failed cases, share findings, and warn about tool limitations
7. Keep ROI and risks in focus. Track where AI has the greatest impact and justifies the investment. Limit the use of AI in critical areas (User data, security) before confidence and the necessary controls

#AI #ML #PlatformEngineering #Software #Architecture #Processes #DevEx #Devops