DevOps Metrics. Your biggest mistake might be collecting the wrong data (Category Management)
For the sake of interest, I decided to read whitepaper Six years ago, in which Nicole Forsgren and Mik Kersten talked about devops metrics. It is interesting that the article begins with quotes from the great
“Software is eating the world.” —Marc Andreessen “You can't manage what you don't measure.” —Peter Drucker And then there is the immersion in the era of digital and devops transformation and the sale of stories to various CIOs that organizations often fail to achieve IT goals due to problems with the speed of software delivery. In order to improve these processes, the authors propose to work on the creation of metrics that will ensure transparency and manageability in this area. For this purpose, they propose to combine 2 type
1) System-based metrics These are metrics based on data from real systems. For example, data on assemblies, auto tests, releases, work in IDE and so on. Consideration of these data requires consideration of
- Completeness Are the data collected from a particular system sufficiently complete to provide the visibility, metrics and reports that are the aim of the initiative?
- Comprehensiveness Is there enough data in all systems to take into account a cross-cutting metric, such as time-to-market?
- Correctness Is the data correlated enough to be correct? Conditionally, you need to be able to match match matching data from different systems. These metrics have steep advantages.
- Precision Data in systems is created with great accuracy
- Continuous visibility Data from systems is available in real time, you can work on the data stream or analyze them after the fact
- Granularity We can look at data from different subsystems or components, or we can go up to the system level.
- Scalability When the data collection system is implemented, it can be extended to all products/projects. But there are challenges.
- Gaining a holistic view It is difficult to collect data from all systems. (tulle)Because we have socio-technical systems, we need information from the eyes of developers about how engineering processes work.
- Capturing drifts in the system When changes in systems, data from them may not be updated, which gives an irrelevant picture
2) Survey-based metrics These are metrics based on surveys of how processes, systems, or people feel:) Such data have the following important aspects: - Cohesiveness Survey data are particularly good for providing a complete and holistic view of systems. - Correctness Survey design and measurement is a well-researched discipline that can be used to provide quality data and an understanding of systems and culture.
These metrics have steep advantages.
- Accuracy properly collected
- A holistic view of the system The answers given by respondents reflect their overall perception of the situation.
- Triangulation with system data Can be combined with system data
- Capturing behavior outside of the system
- Cultural or perceptual measures related to the system But the list of shortcomings is also wide: the accuracy of data is not very large, it is impossible to obtain data constantly. (No one will fill out surveys too often.)The amount of data collected (Few people are willing to complete complex surveys, even rarely.). And if the survey data is used for assessments, the answers in the surveys can be shifted towards the expectations of top management.
In general, the authors conclude that the combination of the two approaches gives an excellent effect. By the way, you can read more about the development of this topic in my post.Why Do Developer Productivity in a Large Company?"
#Management #Leadership #Software #SoftwareDevelopment #Architecture #SoftwareArchitecture #Metrics #Devops #Processes