[2/3] Enabling the Study of Software Development Behavior with Cross-Tool Logs (Category Management)
Continuing my discussion, I’ll cover privacy principles, validation, an example study and the consequences of developing InSession, including later work that used it. The authors followed these privacy principles:
- Collect data only from employees.
- Focus on work tools.
- Do not collect content created by employees.
- Encrypt stored data.
- Make data access auditable.
- Prohibit reporting on individual employees without consent. ||Interestingly, enforcing this principle in practice is extremely difficult.||
- Delete data after a defined retention period of 3 years.
To validate the measurements, the authors compared them with behavioral self-reports, or diaries. In a study with 25 Google engineers who recorded their activities, they compared diaries with session data using PABAK (Prevalence and Bias Adjusted Kappa). Agreement was high for review time (0.81), coding (0.69) and exploration (0.70).
They then examined whether engineers’ readability certification affected review time. Certification establishes familiarity with a language’s idioms and Google’s coding conventions. The hypothesis was that certified engineers would require:
- Less time from reviewers.
- Less time themselves addressing review comments, or shepherding.
The analysis used linear regression controlling for developer tenure, number of reviewers and change size, with a random effect for author identity. Results were:
- For C++, review time fell by 4.5% and shepherding time by 10.5%.
- For Java, shepherding time fell by 10.0%.
- 88% of engineers who completed Java readability agreed that the experience was positive.
The study substantially influenced approaches to developer-productivity measurement. InSession became a basis for addressing longstanding software-engineering questions such as whether types make development more efficient. Google’s use of cross-tool logs to measure developer behavior inspired other organizations to create similar systems.
Later papers mentioning this methodology include:
- “What Improves Developer Productivity at Google? Code Quality” (2022), which used the InSession methodology to examine panel data and establish causal relationships between code quality and developer productivity. I have already discussed it.
- “Predicting Developers’ Negative Feelings About Code Review” (2020), which used InSession data to predict negative interpersonal interactions during review, using the 90th percentile of review and shepherding time.
- “The Pushback Effects of Race, Ethnicity, Gender, and Age in Code Review” (2022), which used the InSession data infrastructure to study demographic factors in code review.
The final post will conclude the discussion with images from the paper.
#Engineering #Software #Bigtech #Productivity #Management #Leadership #Processes