[2/3] What's DAT? Three Case Studies of Measuring Software Development Productivity at Meta With Diff Authoring Time (Category Productivity)

Continue. story About this whitepaper from Meta, whose activities are prohibited on the territory of the Russian Federation, let’s move to the validation approach using DAT (Diff Authoring Time). To this end, the authors conducted additional studies.

1. User experience research Created ground truth dataset by recording real-world developer work Used random sampling to minimize bias Average DAT accuracy is higher 90% compared to real data

2. Large-scale survey Compared DAT with developer estimates (968 unique diff) Built surveys into the Phabricator tool, which is used for the review code. The survey starts immediately after the completion of the diff 3. Descriptive statistics

• DAT covers. 87% of all suitable diffs
• DAT was a stable metric. (used 99Percentennial Winsorized Mean for Reporting) Time Spent by Diff is defined as “averages coding time in a given period by the number of diffs published in that period.” 4. Time series visualization Detailed visualization of how raw telemetry is converted to DAT (The image will be in the final post.) Cross-validation with the authors of the changes (self-developing)

Next, the authors talk about 3 The specific experiments they conducted

1. Typed mocking in Hack The essence of the experiment is to introduce typing into the mopping tools inside the Hack. (internal version of modified PHP). For the experiment, the authors migrated part of the moks into types, and left part as is and then compared DAT when creating diffs in different parts of the codebase. The experiment showed how language capabilities with specific indicators of productivity

• 14% improvement in DAT: First quantitative evidence of the effect of typing on productivity in an industrial environment Statistical significance: p < 0.001 full-size

2. Auto-memotion in React compiler The authors refined the React framework for auto-memoization and then conducted an experiment where they compared DAT when creating diffs with manual and automatic memoization. They used a mixed effect model to account for confounders through a regression model. For nonrandomized data, they used Wasserstein distance to measure the true difference between grappumi.

• 33% improvement in DAT: Significant increase in efficiency when using automatic memoization

3. Analysis of the effectiveness of code reuse I found this study the most interesting, as the authors assessed the impact of cross-platform technologies. (The kid had it. React). However, the analysis had to use counterfactual analysis to estimate the hypothetical development time without reusing the code. As it turned out,

• Cross-platform gives more than 50% improvement over development without reuse That’s thousands of hours of annual DAT savings through code reuse frameworks.

As a result, this approach to using the DAT metric led to the following effects: Transition Infrastructure teams to a culture focused on a/b experiments Data-based decision making – DAT is used to plan and prioritize development DAT and experiments align approaches between product and infrastructure teams

If we talk about the further plans of the authors of the study, they plan to expand. Horizontal - Addition other than diffs and other artifacts (documents and tasks). This will create a common framework for measuring activity time for experiments. Vertical - Support for more tools (Different IDEs and other tools). This will allow you to focus less on heuristics and more on accurate measurements.

#Engineering #Software #Bigtech #Productivity #Management #Leadership #Processes