[1/2] Measuring the Impact of AI on Developer Productivity at Meta (AI column)

Excellent. speech from the guys from the banned Meta in Russia, which I have reviewed 3To better understand all the details. Performed by Payam Shodjai (senior director of product management) Pavel Avgustinov (techlead) Meta’s developer productivity areas and they told a bunch of interesting approaches + technical details about how they approach this issue. After this talk, I got a much better understanding of the concepts from the article "What's DAT?" Three Case Studies of Measuring Software Development Productivity at Meta with Diff Authoring Time (see. my analysis).

The main idea of the report revolved around the question “how to measure the impact of AI tools on real engineering productivity in a large company?”. And the report shows that Meta has built a comprehensive set of metrics, compiled comprehensive telemetry, learned to account for complex operations in version control systems. (This was promised to scour the new whitepaper, as the basic git can not do the same. Sapling Meta). Next, they looked for direct correlations between AI use and classic productivity/business value metrics. This wasn’t about standard code completion, but a broader set of AI tools across the SDLC.

To put it simply, the guys showed that without good instrumentalization, the conversation about AI ROI quickly turns into opinions and feelings. Therefore, Meta does not rely on surveys as a single source, but on a bunch of telemetry, diff-metrics, behavioral data and verification of this data in real work sessions. This is consistent with Meta’s Diff Authoring Time system. (DAT)It combines privacy-aware telemetry from IDE, OS and version control and is then validated by observational research, surveys and visualizations.

1Smart Classification of Pull Requests with LLM For a long time, the main metric in the company was DDM. (Diffs per Developer per Month – Number of Diffs/Committees per Developer per Month). However, with the introduction of AI, this metric became vulnerable to the Goodhart Law: developers could artificially inflate the performance by cutting tasks into small commits, and the AI generated a lot of boilerplate. (template). To measure real business value, Meta implemented the Feature DDM metric. To do this, they use LLMs that automatically analyze the contents of each Pull Request and classify it into: Feature Diffs: Code that creates new product value for the user. Non-Feature Diffs: refactoring, configuration updates, tests and documentation. AI assistants are evaluated precisely by how much they increase the number of product diffs.

2Symbol telemetry and internal instruments (Devmate) Meta does not rely on basic version control statistics. To collect data, they use their own internal AI tool – Devmate. (customization). The main nuance of their telemetry - symbol tracking (character-level tracking). The system knows exactly the origin of each character in the code. Meta can tell with absolute accuracy what percentage of characters in the final pool request was printed manually by the engineer, and what percentage was generated by the neural network and left unchanged. Separately, it is mentioned that Sapling allows you to smartly track the transition between commits and more correctly consider how code is transformed using edit/rebase and the rest, as well as by whom. (human). In addition, telemetry tracks bottlenecks throughout the cycle. (SDLC): time of finding a task for code review, approval cycles and passing automatic tests.

Next, post I will tell you about the main insights that the authors shared and which are really interesting.

#Engineering #Software #Bigtech #Productivity #Management #Leadership #Processes #AI #ML #Architecture #DevEx #DevOps