[1/3] What's DAT? Three Case Studies of Measuring Software Development Productivity at Meta With Diff Authoring Time (Category Productivity)
For this interesting article 2025 A year ago from the company "Meta" I came across while preparing a review of the article.Enabling the Study of Software Development Behavior With Cross-Tool Logs" (my review: 1, 2 and 3). And although the activities of the organization "Meta" is prohibited in the territory of the Russian Federation, but to read their engineering whitepaper is quite interesting. The main idea of this study is to measure the productivity of software development using the Diff Authoring Time metric. (DAT)This is the developer’s time to make changes to the code. (diff)Which in essence resemble MR. (merge requests). This time is collected using a telemetry system integrated with the version control system, IDE and operating system.
The key value of this research to Meta is Moving from intuitive assessments to a scientific approach to measuring productivity based on DAT DAT measurements are not aimed at evaluating the performance of specific developers, but rather at assessing the impact of tools and processes on productivity. This approach makes it possible to conduct a/b experiments to change tuling and processes - according to the authors, they have already run it on. 20 Three of which are described in the article. It is important that some experiments can be rolled with granularity at the diffs level. (Experiments that are transparent to engineers)and some at the level of a cohort of engineers (those where changes are clearly visible and do not allow different diffs to change approach, such as switching between IDE versions)
Interestingly, the authors claim something in the style that their research, for the first time in the industry, provides quantitative evidence of the impact of various development tools and techniques on productivity in a real corporate environment. Apparently, they haven't read the Google guys' research, like the one I mentioned above. 2020 year:)
If we talk about the structure of the model, it consists of the following parts: 1. The basic algorithm of accurate comparison Tracking activity in the IDE: The system captures work time in an integrated development environment Integration with version control system: Links temporary sessions to specific commits via Sapling (Meta version control system) Left shift algorithm: Each CHx commit is assigned to an IDE session(x-1)which precedes it. 2. Additional heuristics Anchor Sessions: Captures activity prior to exact matching to more fully cover development time Extreme case handling: Automatic checkout exclusion and filtering of irrelevant activity 3. Telemetry system with privacy protection OS-level telemetry: Monitoring activity at the operating system level Tool integration: Plugins for VS Code, Sapling and other development tools Non-overlapping dimensions: DAT ensures that time between different diffs does not overlap for a single developer
As a result, we get the following key properties of the model:
- Non-overlapping: DAT(D123) ∩ DAT(D987) D123 and D987 are two different diffs.
- LimitationsDAT may not exceed 24 hourly
- Aggregability: Unlike other metrics, DAT can be summed up correctly.
Continuation of the review in next post.
#Engineering #Software #Bigtech #Productivity #Management #Leadership #Processes