DORA Methodology

Igor introduces himself as an external expert and consultant who helps large companies develop engineering culture, processes and practices; he has followed DORA since the team appeared — ten years, ten reports and, per the 2024 report, over 90,000 industry respondents. Alexander recalls last year's episode on Accelerate: the 2017 book summed up the first five years of the survey and offered a scientifically grounded model of how DevOps affects organisational performance — for the industry that was the move from what we believe to what we can measure. Back then the focus was the results; now it is the machinery under the bonnet. The DORA core model, in Igor's words, is simplified and rather a marketing device meant to attract attention; its current version 2.0 links capabilities, performance and outcomes — engineering practices, metrics of speed and reliability, and results for a company, a team or an individual engineer.

The model is developed by the community: about a year ago the discussion was opened on GitHub, and Igor turned up there proposing to map onto it the three ways from the DevOps Handbook — practices that speed up delivery, improve feedback and build engineering culture. The Fast Flow and Feedback Loops categories appeared thanks to his comment; the terms were worked through as well — what counts as maintainability, what counts as a small change — because in research terminology must be agreed at the start. DORA was founded by Gene Kim and Jez Humble, authors of well-known books, while the research side was brought in by Nicole Forsgren, a scientist who knew the methods the founders lacked. In 2018, after the book came out, Google Cloud bought DORA, and it is now a small team inside Google Cloud; from the following year, as Alexander recalls it, the partner from the configuration-management world left the joint report; Igor adds that the partner carried on with its own research, now focused on platform engineering.

What pushed Alexander into the methodology were the phrasings in the 2024 summary of results: sentences of the shape "a 25% increase in adoption leads to…" struck him as an unusual statistical construction. He complains that methodology and the way numbers are computed get little attention, which leaves room for manipulation — as in papers where sixteen developers are handed tasks and sweeping conclusions follow. The methodology does describe continuity with previous years, so trends can be compared, plus questions about capabilities and outcomes and questions that filter out a third variable. But Igor sees a missing step: research starts not with questions but with a goal, a model and hypotheses. DORA publishes the questions and answer options on its site, yet the hypotheses and the big model are absent — the research comes out half-open and cannot be reproduced. Since 2023 one big model has given way to small ones: a single capability plus the outcomes of interest, for generative AI, platforms or workflows.

The respondent's experience is a topic of its own. The longer the survey, the worse the completion rate; wording has to read unambiguously, or the answer depends on how the person understood the question; and the cognitive complexity of technical constructs makes simple questions hard to write. Igor points out that everyone is tired of surveys and a respondent needs value — open data, say — while internal surveys should not ask what can be loaded from other systems: tenure, role, team. The answer options always include "I don't know", and by his reckoning more than half of DORA's questions use a Likert scale. Terms have to be prepared in advance through books and white papers, otherwise release frequency, change lead time and MTTR drift apart: released to which environment, MTTR for a service or for a pipeline. Alexander adds that DORA set the fashion in terminology, and the Likert scale was rarely seen in IT surveys before it: Accelerate explained why stating agreement is easier than picking a number or a range.

A sample must be representative of the population, and Alexander recalls the 1936 case: a large magazine sent its poll to roughly ten million people and predicted a comfortable Republican win, yet the Democrat won — the magazine's readers were people of high income. The Gallup institute took two thousand people and got it right. DORA recruits organically, through communities, and allows the link to be shared — which makes a snowball effect and skew from one powerful community easy to catch — and as a second move writes to under-represented participants. Igor agrees, with an example from another report: State of Agile runs at a SAFe conference, so SAFe predictably tops it; he suggests inviting people personally and giving them the block of questions matching their area. Alexander objects that this is exactly how bias is manufactured — like polling architects at an architecture conference and concluding that the whole industry has an architecture repository and fitness functions. DORA itself split the survey into three branches and assigns respondents at random, preserving the legacy link between the DORA metrics and outcomes.

Constructs are checked for internal and external validity. Alexander explains the internal kind through the elephant and the wise men: if someone credits the elephant with an eagle's wings, the construct does not assemble. External validity is about links to reality, and the 2024 report has places where it did not hold: explanations have to be invented, such as a vacuum hypothesis about meaningful tasks, or why a platform drags stability down. A construct that does work is Ron Westrum's culture model: the directive kind, where the messenger with bad news is killed and information about problems does not travel; the bureaucratic kind, living by the letter of its rules; and the generative kind, where problems are taken apart down to root causes — the only place SRE practices really work. The sample is not large — around 3,000 people took the 2024 survey — so precision is squeezed out with leave-one-out cross-validation and the Watanabe–Akaike information criterion. The main objection to DORA, in Igor's words, is precisely about causality: Alexander, whose unit at T-Bank owns the company's A/B experimentation platform, accepts correlations, but no real experiment can be run inside a survey, much as an astronomer cannot light a new star.