How and Why to Measure Engineering Productivity

The starting observation is plain. In a large technology company the effectiveness of engineers contributes heavily to the effectiveness of the business, yet aggregate indicators describe the situation as a whole and give no way to isolate the contribution of individual parts of the organisation. To narrow the subject, the author splits processes into three kinds: Run, the recurring operational work; Change, building and evolving products; and Disrupt, internal startups and shifts of business model. Everything that follows concerns Change, and specifically the delivery part of it.

That framing also sets the boundary of applicability. The frameworks under discussion grew inside product companies and sit badly on consulting or outsourcing arrangements. The author is equally explicit about what was wrong with earlier attempts to measure development work: they counted output instead of outcome and looked at the individual instead of the whole system, which produced local improvements with no effect on the end result.

DORA came out of the research of Forsgren, Humble and Kim and the 2018 book Accelerate, after which the work moved under Google. Its four metrics fall into two groups: delivery tempo, covering deployment frequency and lead time for changes, and stability, covering change failure rate and time to restore service. Behind the model sits a large body of survey work whose analysis confirms a positive correlation between delivery tempo and service stability; by 2024 the core model had absorbed ideas from neighbouring approaches.

SPACE, published in March 2021 by Nicole Forsgren together with Zimmermann, Houck and Butler, breaks productivity into five dimensions — satisfaction and well-being, performance, activity, communication and collaboration, efficiency and flow — and reads each of them at three levels: the individual, the team, and the end-to-end process. DevEx, by Forsgren, Noda, Storey and Greiler, compresses the picture into three areas: feedback loops, cognitive load and flow state. Google's QUANTS was assembled by a team of engineers and social scientists and rests on the GSM method, moving from goals to signals to metrics.

Some signals can only be collected by asking people. The DevEx authors say so directly, and the 2024 continuation of the framework rests on statistical analysis of surveys run on the getdx.com platform. Other signals can be pulled straight out of working systems, and both schools have commercial products behind them: Cortex builds team scorecards on top of DORA, SPACE and DevEx, DevEx 360 operates as a survey platform, Code Climate leans on DORA metrics and goal reporting, and Pluralsight Flow tracks cycle time and pull request activity. Etsy, Dropbox, eBay, Amplitude, Monzo, P&G and DHL are named among DevEx users.

T-Meter, run as a product by Pavel Akhmetchanov, belongs to the second school: templated wiki spaces and dashboards built over Jira with custom task workflows. A team sees its delivery rate, the time tasks spend in each status so bottlenecks become visible, backlog dynamics and development cycle time. A leader sees the same picture rolled up across teams, with lead time at the 85th percentile, backlog-to-delivery and work-in-progress-to-delivery ratios, and predictability expressed as the 85th percentile divided by the 50th. The author warns twice over: mapping the status model properly is not trivial work, and if such metrics are imposed from above the measure turns into the target and everyone starts painting dashboards.