[2/2] AI in SDLC: the way from assistants to agents (AI column)

Continue. story About my report, I will share my thoughts on how to approach AI-fication development that is based on whitepaper. Measuring Developer Goals. Understanding and effectively measuring goals is critical to improving the developer experience and improving their effectiveness. To answer questions about productivity, it is more convenient to link measurements not to specific tools, but to the goals that developers set for themselves when using tools. This allows you to answer questions similar to those above, keeping metrics user-centric rather than a tool. Detailed analysis bloggingand also podcast Research Insights Made Simple, where we discussed this article with Sasha Kusurgashev, my colleague who directs the development of Spirit. (Our internal development platform). In general, the whole series of articles about "Developer Productivity for Humans", which I have analyzed in two posts, can be useful. (1 and 2). If you tie these ideas to the implementation of AI in development, then it is clear that you need to focus on the main jobs to be done scenarios that are massive / problematic - you can get maximum effect with them. And it is worth starting with AI-assisted things, and as assistance improves, move towards delegating the entire scenario to the agent. (However, here you will have to work with the preparation of the context, setting up evaluation, changes in the processes of people’s work.). And as you change, you need to be able to measure the effects of those improvements so that you can show the results to top managers. (||And they like numbers.||).

Measurement of productivity (how and why) I already am. told Before, but there were classic DORA, SPACE, DevEx. And lately, I’ve been watching DX’s performance measurement platform, which was founded by those who developed previous approaches. These guys built a system with surveys and integration with systems like Jira, Wiki, git, CI/CD, ... In general, the guys came up with their own framework DX Core 4 Measuring Engineering Productivity (centimeter my textual analysis + parsing in my podcast with Zhenya Sergeev of Flo)This year, they expanded it with a branch to measure the effectiveness of AI assistants and agents. (centimeter my textual analysis + parsing in my podcast with Gene). In fact, the effectiveness of AI assistants and agents can be measured in three ways.

1. Utilization Monitoring the implementation and use of AI tools (DAU, MAU, % generated code, % PR with AI assistance). This is usually the beginning of measurements, as these indicators are easier to measure than those in the following paragraphs. (Impact, Cost)
2. Impact Measuring the real impact on productivity (Developer time savings, PR throughput, percieved rate of delivery)
3. Cost tracking costs and net income The guys at DX have benchmarks on these metrics that they provide to DX platform customers.

If we talk about our approach in T-Bank, we are able to measure the first level of Utilization, and we have implemented a SPACE framework for assessing developer productivity. This allows us to move towards Impact evaluation. By the way, about our framework SPACE guys told on IT Picnic and here's the analysis This talk. But if you do not have tools for measuring developer productivity in the company, then do not be sad. The level of recycling can be measured relatively easily, and more you may not need - now the development is changing revolutionaryly through the use of AI assistants and AI agents, which means you can not invest a lot of effort in measuring the old approach to business, and experiment with the new. Conventionally, you should not measure the wooden wheel of a stagecoach, if we already have it replaced by a metal wheel of a car:) Well, if you want benchmarks, you can participate in our Large study of AI in engineering culture of Russia.

#Software #Engineering #Productivity #DevEx #AI #Management #RnD #Leadership #Economy