Measuring Productivity: All Models are Wrong But Some are Useful (Category Management)

This one whitepaper Google researchers Sera Jaspan and Collin Greene present a text version of their DPE Summit speech (Developer Productivity Engineering). The title of the article contains a famous quote. George Box. “Essentially, all models are wrong, but some are useful,” which applies perfectly to measuring engineers’ productivity. They talk about the principles and methods that Google has developed to overcome the limitations of models and gain valuable insights. This article continues the "Developer Productivity for Humans" series in the IEEE journal, all articles from which are reviewed in the journal IEEE. postLet’s talk about this white paper.

1. Measuring developer productivity is the construction of models where it is necessary to accept that no single measurement approach can perfectly capture the complexity of software development. The authors argue that productivity models are often “dangerously selective” because they omit important aspects of developers’ work, leading to incomplete or distorting estimates. It is important to understand that effective productivity measurement requires an integrated approach that covers several aspects of work, rather than relying on primitive metrics such as the number of lines of code or the frequency of commits.
2. Often, such a comprehensive approach involves compromises between different aspects of software development, for example, it is easy to speed up velocity if you throw out the stage of code review and testing. For us, this means that productivity should be seen as a balance between several factors, primarily speed, convenience, and quality, rather than a single measurable outcome. At Google, this tradeoff is represented as a triangle: speed, ease, quality.
3. Researchers at Google also look at sociological aspects, considering them in conjunction with technological ones. It is this approach that has given the title to the entire series of articles where measuring productivity should take into account the complex, creative nature of a developer’s work, rather than treating it as a purely mechanical process.
4. This study provides a structured model for building productivity measurement systems that avoids common mistakes. It seems that the authors use the approach of Goals/Signals/Metrics. (GSM)Which I'm talking about. wroteWhen I read “Software Engineering at Google,” they didn’t say it.)
5. The authors consider two opposite factors: the desire for simplicity. (parsymony) Avoiding dangerous selectivity (worrying selectivity). Parsimonia encourages the inclusion of fewer components for the sake of simplicity, and worrying selectivity requires the inclusion of more aspects to cover all important aspects of productivity. The study advises leaning toward fuller coverage rather than excessive simplicity.
6. The authors combine qualitative and quantitative methods in their methodology using data from tool logs and surveys. This blended approach reflects their understanding that developer productivity cannot be fully embraced by metrics alone or purely subjective assessments. Instead, they suggest combining different types of data to better understand the picture. Interestingly, the structure of their model takes into account the effect of the measurements themselves on behavior: what is measured often begins to affect employee behavior, sometimes contrary to original goals.

In general, the authors propose to measure the productivity of engineers, understanding the limitations of measurement models, and to use an integrated, multidimensional approach, fixing tradeoffs and validating conclusions using different methods.

#Engineering #Software #Bigtech #Productivity #Management #Leadership #Processes