Skip to content
#AI

Measuring AI in SDLC: From Adoption Metrics to Business Value and User Outcome (AI column)

#AI #Metrics #Engineering #Management #Software

About a week ago, my colleague Anna Gromova performed at Yandex AI Dev Day. She directs product analytics for internal platforms, SDLC processes, and AI tools, which is why she was the best person to talk about it. Speech It was quite short and had a similar structure.

1️ What is SDLC and why is it measured? If Software development life cycle is the process of delivering code to production. To assess it, T-Bank used three frameworks widely known in narrow circles:

  • DORA The oldest framework that evaluates the speed and reliability of delivery (right here told framework 2026 the fifth metric year)
  • SPACE - a framework that takes into account not only the conveyor, but also satisfaction, the state of the developer's flow, and a lot of other things. (here speech Sashi Kusurgasheva from summer 2025 The year he told us how we were preparing this framework.)
  • DevEx Framework with a focus on cognitive load, context switching, feedback loop (here speech Kalugin's wows about how we prepare DevEx) Using the thoughts from these reference frameworks, we at T-Bank have builttree" (||But it looks more like a network.||): upper level - DORA, lower - DevEx, SPACE - somewhere in the middle + it is based on a semi-annual survey of thousands of engineers. By the way, all these approaches. DORA, SPACE, DevEx I've told you a lot before.

2T-Bank’s AI Assistant and its Acceptance Level (adoption) Internal AI assistant is implemented in IDE, Jupyter laptops, BI systems, GitLab, observability and search. Modes of work in IDE: tips, chat, agent mode. Adoption metrics

  • IDE (from GitLab users) - ~70–75%
  • Analytics tools - ~75%
  • All surfaces total -57% At this level of use, we can already draw conclusions about the impact of AI on SDLC metrics.

3From adoption metrics to user scenario n Previously, only adoption was considered: new users, outflow, share of generated artifacts, “meaningfulness” of use. (regeneration, resetting context), likes/dislikes. In 2025–2026 year-round pyramid:

  • Upper level - SDLC business metrics (time to market, reliability)
  • Medium. Closed user scenarios (generation of unit tests, assistance in code review, etc.)
  • Lower Technical assistant metrics (speed, model version) This allows you to simultaneously respond to business issues and develop an AI product.

4The role of telemetry The key thesis: **First determine the metrics, then decide which logs to collect.**Not the other way around. This reduces squats around telemetry and focuses on the real goal. Uniform data is important to the results of different teams (AI team, CICD team) They were consistent with each other.

There are more numbers in the report about the results of implementation in different scenarios, but I will leave this to those who are interested and ready to watch the original video. In general, the presentation is useful and interesting, and it also demonstrates that in Russia we are close to SOTA in terms of methods for measuring the impact of AI on development. (But we have a lot of ideas about how we can approach SOTA globally.).

#AI #Metrics #Engineering #Management #Software