Agents Write Code: Measuring the Effect on a Team

Vladimir Dolgov arrived at developer tools through large corporate projects. While working on customer systems, he had detailed information about customers and operators but much less visibility into engineers’ daily work. Trackers and repositories showed only parts of the picture. AI widened that gap: a manager’s accumulated experience no longer reliably indicated how long a new task would take.

Vladimir now builds DevOptify and uses it with his own team. His tools include Cursor, Codex, and Claude; Alexander adds his experience combining models and keeping instructions portable. The point is to match tools to work. A subscription used by an individual engineer has different economics from a commercial product that calls models through APIs. A useful personal setup does not automatically make an equivalent product commercially viable.

Vladimir describes successive attempts to measure AI adoption at Sber. The first metric was the share of generated code. Management asked whether that figure justified reducing headcount. It did not: generating program text covered only one part of the work. The team then measured PRs per engineer and compared groups using the tools with groups working without them. An increase still did not make the benefit obvious to business stakeholders.

The next agreement concerned completed tasks. Vladimir does not present that measure as universal: the relevant outcome is one the business recognizes as useful. Different management levels need different measures and checks. More code needs to be considered alongside defects and rework; more tasks need task-size context. Otherwise, splitting work into smaller tickets can improve a report without changing what reaches the user.

Alexander recalls a team whose completed-task count increased substantially. The explanation lay in the work mix: it had first built a shared platform, then used that platform to deliver many repeatable product changes. The chart alone could not attribute the entire increase to process improvements. Even a conscientious team can mistake a change in task composition for evidence of greater effectiveness.

Vladimir adds the product lifecycle. Teams exploring a new direction, expanding a product, and maintaining a product nearing retirement serve different purposes. The last team may need to support customers reliably and fix defects rather than release substantial new functionality. Comparing teams or individuals without that context creates misleading expectations. Understand the purpose of the work before deciding what the numbers mean and what action they justify.

Vladimir finds it useful to think of an agent as a digital employee. It needs general and domain knowledge, tools, a usable environment, and limits on its authority. Explaining rules and providing documentation resembles onboarding a person. Autonomy at each stage depends on how much the team trusts a particular agent and its output.

This makes communication relevant even to junior engineers: they need to state a task clearly, identify why work has stalled, and give useful feedback. Telling an agent to try harder does not explain what to change. Meanwhile, agent output can exceed human review capacity. Vladimir says his team now reserves human code review mainly for complex tasks. That is an account of the team’s practice, not evidence that review is unnecessary for every product.

The DevOptify demonstration uses the current work of Vladimir’s team. Planning accuracy is a central measure. Initially, people estimated tasks using older assumptions and finished much sooner. Confidence in the tools then pushed estimates in the opposite direction: tasks began taking longer than expected. The team is still learning to plan predictably under these conditions.

The demonstration also distinguishes exploration from delivery. Much of the AI activity in the example supported research, so it could not be read directly as released functionality. Some engineers run many tasks in parallel, while others work more sequentially. Vladimir suggests investigating which practices colleagues could learn and who needs help, while retaining the differences between their actual assignments.

The platform associates events from IDEs, browsers, and command-line tools with tasks and groups them into work sessions. That measures something different from a ticket’s time in an “in progress” status: several open tickets do not mean someone is actively working on each one. Rules for grouping events and stopping the clock during gaps determine what the resulting time measure includes.

Vladimir asks the system about repeatedly reopened tasks, changes in delivery time, and the balance between fixes and other work. The proportion of defects by count differs from their share of effort. Night and weekend activity can prompt a workload conversation, especially when a capable engineer receives too many assignments. An experiment using changes in prompt tone is likewise presented as a reason to investigate; Vladimir explicitly calls the approach exploratory.

Token consumption can help examine tool choices. Vladimir describes using a large model for bulk identifier replacement when a standard IDE operation would have sufficed. High usage alone does not establish poor work, however: difficult tasks may need stronger models. A leaderboard based on tokens consumed is even less informative about value. Resources can be spent on activity unrelated to company tasks.

Discussing adoption, Vladimir describes cloud and on-premises versions, MCP integration, and collection limited to approved work projects. Employees can see the same picture as managers and challenge its interpretation. In one example, an engineer showed how many unplanned tasks a product owner had added, explaining why the original plan was missed. Shared data can support renegotiating commitments instead of automatically assigning blame.

At the time of the conversation, Vladimir is entering large Russian pilots. He describes establishing a baseline, drawing conclusions and changing the work, then observing the outcome. He estimates that observation may take about a month before the intervention and another month afterward. Demonstrating a dashboard alone does not establish that the product is useful.

Alexander closes by returning to a sequence for assessing AI: adoption, changes in specific work, and then economics. Usage data can start an investigation, but managers still need to connect it to outcomes and costs. The conversation’s practical conclusion is that analytics earns its place by helping someone choose an action, explain it to the team, and check what actually changed afterward.