SDLC with AI look through metrics - Anna Gromova @ AI Dev Conf (Category AI4SDLC)
Watched May. speech Anna Gromova from T-Bank from the AI Dev Conf conference. Anya and I often discuss AI performance metrics at work, so it’s especially helpful to me that there’s a record that can now be referenced: it’s a relevant and in-depth analysis of how we approach this topic at home.
The main idea there is very practical: AI in development can not be properly evaluated if you do not understand the process of code delivery. The number of licenses, MAUs, assistant requests, or strings generated can be a useful adoption signal, but it does not answer the main question: has SDLC become faster, more reliable, and cheaper for a real production result?
Anya begins with frameworks that many know individually, but rarely put together.
- DORA These are conveyor metrics: lead time for changes, deployment frequency, recovery time, change failure rate; the report discusses unplanned work next to them. That is, how quickly and how consistently the team delivers change to production.
- SPACE and DevEx Add the human side: satisfaction, communication, flow, cognitive load, feedback loops. Because development is not only a pipeline, but also the people inside it: waiting for review, switching contexts, going to meetings, sorting out a new project and looking for access. By the way, I have already talked in detail about all these approaches. DORA, SPACE, DevEx.
I like that in her speech Anya did not dwell on theory, but showed real metrics: onboarding time, time to the first MR, pipeline time, testing time, review time, rework, MR size, the share of falling pipelines, incident recovery, developer surveys. So it's not that we don't have a single magic metric for productivity, but a map of the process by which we can look for bottlenecks.
For example, if lead time has grown, it may not be because developers have become slower. It may turn out that people take too long to get accesses, documentation is bad for onboarding, new end-to-end tests have slowed the pipeline, or MRs have become too large and hang on review.
And here AI becomes interesting not as a “coding accelerator”, but as an intervention in specific areas of SDLC.
AI review can reduce the wait for the first reaction and some of the routine iterations by style, typos and simple bugs. AI in debugging problems can more quickly reduce logics, code, and symptoms to a hypothesis about a key cause. The generation of unit tests closes the boring but important work, which is often not reached, and through this affects the quality of the pipeline and the change failure rate.
But an important caveat: if you speed up only code writing, the bottleneck will simply move on. First, the senior does not have time to check the generated code. Then come the flak tests and CI/CD. Then to test environments, release process or platform limits. AI doesn’t fix a weak engineering system by itself, it often just highlights its limitations.
According to Anna, in T-Bank, AI ambassadors saw a decrease in median merge time. 12% and lead time 30%. It’s not a universal bench and it’s not a “just add” promise. ~~water~~ AI and you get a minus 30%”: the report separately states that the features of the repositories, coherence and complexity of the code remain behind the brackets. But as an example of the right conversation about the AI effect, this is valuable: not “people started pressing the button more often,” but “what happened to the delivery process.”
It’s clear from the report that meaningful AI implementation requires a baseline prior to launch, a regular collection of metrics, linking quantitative data to developer experience, and understanding what part of the process we really want to improve. More importantly, don’t fall in love with one metric. Adoption is necessary, but by itself it proves nothing. Throughput is necessary, but without quality/risk, you can simply produce problems faster. The economy is needed, but without an engineering context, it’s easy to start optimizing the value of tokens instead of the value of the result.
Practically, this means that before you argue which AI tool is better, you should honestly describe your SDLC. Where do you waste your time? Where does the feedback loop break? Where does review turn into a queue? Where Does CI/CD Eat the Benefits of Code Generation? Where do developers feel the cognitive load and the metric graph shows the same? Then the conversation about AI becomes an adult. It’s not “magic accelerated development,” it’s “we changed a specific piece of the engineering system, saw the effect, tested the risks, and figured out where the next bottleneck is.”
#AI #AI4SDLC #Engineering #DevOps #Management #Metrics #Software #Processes