[2/2] Measuring the Impact of AI on Developer Productivity at Meta (AI column)
This post continues parsing fine speech The guys at Meta have insights that they shared with the audience.
1Users of DevMate (An internal version of Claude Code or Cursor) above average Meta observed approximately 6–12Percentage of DDM (Diffs per developer per month). It doesn't sound like a revolution in 2 But for a large engineering organization, such an increase already looks significant, especially if it is consistently reproduced on large amounts of internal data.
2The effect of using AI was uneven. While AI generates conditional 10–30% changes in the diff, the gain in time is almost not visible. And noticeable improvements appeared when AI made more 60% code in diff. This hints that the maximum payoff is not when AI is used “a little bit everywhere,” but when the task and workflow actually allow the tool to hand over a large piece of routine work. That is, AI pays off more in tasks where you can delegate a significant amount of mechanical code, not just tips on trivia.
3Senior engineers used AI more effectively than junior engineers, although juniors could use it more often. This is an important anti-intuitive point: more frequent use does not equal greater effect. It seems that the winners are those who better define the context, know how to check the response of the model and understand where AI is worth trusting and where it is not. As a result, on average, signor diffs contain a significantly higher percentage of AI code than junes diffs. The thesis of the guys is that experience, architectural thinking and the ability to formulate accurate instructions greatly enhance the effect of AI - Signor seems to just give TK not juna, but models. 4After the introduction of AI, there may be a drawdown first, and only then productivity growth (J-curve of adaptation). According to Meta, there is a drawdown in the productivity of order 15%: people learn to prompt, recheck the code, change the usual process. After adaptation (several months) begins a stable plus to DDM and a reduction in coding time per diff, which gives the final 6–12Percent growth in output. The practical point here is simple: if a company looks at the AI effect only in the first weeks after rollout, it may mistakenly decide that the tool doesn’t work.
5Telemetry has shown that engineers are less likely to sit in chat rooms and documents because answers and context are received directly in the IDE via DevMate. Because of this, the time “coding time per diff” formally increases, but the authors consider this a good signal: less twitching on links and messengers, more focus in one tool. 6️ Not all teams win in the same way – teams with a high ML/speech ratio show less DDM growth because it takes a lot of time on laptops, experiments and analytics, which are poorly reflected in diffs. Also, DDM is very “noisy” from holidays and external factors, so Meta does not use it head-on as a KPI for people, but only as an aggregate product metric of the impact of AI tools. To sum it up, I can recommend this video and the accompanying whitepaper to anyone involved in measuring the effect of AI in development – a good methodological and practical approach to this complex topic.
#Engineering #Software #Bigtech #Productivity #Management #Leadership #Processes #AI #ML #Architecture #DevEx #DevOps