Measuring the effectiveness of software development tools and practices (Category Productivity)
I often talk about the productivity of engineers, as well as the approaches of large companies. (look Google articlesor Articles banned in Russia Meta)and also about how we Inside the T-Bank, come on. to this. But today I wanted to talk about Amazon The authors propose a method for measuring the economic effect of development tools and practices. They offer to look at a single metric CTS-SW (cost-to-serve software) In fact, how many resources are spent on delivering a unit of software to the client. This metric should link the usual engineering metrics in the spirit of DORA / SPACE with a clear business result: saved engineering capacity or money.
Interestingly, we also have similar approaches to calculations within T-Bank, which are based on the unification of development processes and the allocation of the level of tasks in teams, which bring clear business results. But let’s go back to the original article and see how the authors’ approach works.
1They take “software delivery cost” as the final metric, rather than trying to estimate each development step in detail. The authors say that classic activity-based costing for software is too fragile and therefore they simplify the task to “input costs → unit of result”, and then look for drivers of this cost throughout the life cycle: coding, CI / CD, planning, incident management, maintenance, search for information, etc. The unit of result is also selected for the architecture.
- Microservices.
- Monolith - stitched in master code
- Libraries - Commites
2Next, they build panel data (mix of factors + time) They use linear mixed models. They have telemetry on thousands of "two-pizza teams" over five years, and they model CTS-SW through developer time on the depot. They need linear mixed models because they simultaneously capture the overall effect of factors and differences between teams. So they found the main CTS-SW driver candidates:
- Team velocity (How much rev code a team merjits per week per engineer) - the strongest predictor.
- Delivery health. (e.g. Rate Rolbeck)
- Paige's on-call engineer.
3Moving from Correlation to Causal Inference After finding effects, the authors look for an opportunity to experiment. Such a natural experiment for them was the introduction of genAI tools, in particular Amazon Q Developer. To assess, they build panel regression with dynamic two-way fixed effects: taking into account constant differences between commands, common time effects, past speed lag and time-varying covariates such as the proportion of the team using Q Developer, rollback rate and manual interventions. The goal is not just to see the correlation, but to isolate the instrument’s causal contribution to CR velocity and deployment velocity.
4Add “insurance” against overestimation of the effect of AI The authors separately recognize that simply adding up the effects of different experiments can overstate the real impact. Therefore, they are developing a baseline model to normalize the impact estimates of AI tools.
This is an interesting approach that allows Decide which dev tools and AI tools to scale based on measurable effect Prioritize CI/CD automation and reliability practices because they are linked to the best CTS-SW. Create conditions for high team velocity through onboarding, but use this as a team metric, not an individual metric. Carefully interpret “useful” signals
In the future, the authors want to expand the model to compare architectural solutions and make recommendations.
#AI #Engineering #Management #SystemDesign #Architecture #Processes #Software #Software #Agents #Economics