Measuring AI Code Assistants and Agents

Alexander opens with a fresh article from the DX platform — not a white paper, as he corrects himself, but an article published literally this week or the last. He describes DX as a large developer-productivity measurement platform whose contributors include co-authors of DORA, SPACE and DevEx, bringing those approaches together in DX Core 4. DX combines research and frameworks with an accumulated dataset used for benchmarks in its commercial platform.

Evgeny Sergeev, Engineering Director at Flo Health, builds internal products that help the company create, review and release content faster. Development, Alexander explains, is a convenient testing ground partly because it is well digitised: metrics, tooling and plenty of checks catch whatever assistants and agents have done, so trying things and collecting feedback is easy.

The article has several parts: the framework itself, a separate block on rolling metrics out — a sensitive matter where a mistake lowers productivity instead of raising it — and a pointer to the bigger picture, in which measuring AI is only part of measuring productivity. From the roundtable Evgeny brings the market's main worry, return on investment: AI is still presented as a copilot and a second pair of eyes for the engineer, and it can draw on an existing base of linters and checks, with context supplied through tools and prompts.

Industry benchmarks help interpret the numbers. A hypothetical "80% adoption" does not by itself show how a company compares with others; DORA used groups such as elite companies for that comparison.

Alexander says his team agreed to spend the summer designing their own survey by the canons of the last part of Accelerate — wording of constructs, a Likert scale, cross-validation of questions — and, if it works out, to launch it as a benchmark of the Russian industry closer to the end of summer: there is plenty of hype and plenty of numbers around, but how they were measured, and what follows, is usually unclear. What Evgeny likes is their credibility in methodology: they might push the approach into an industry standard that could be run in another locale.

Evgeny reports an observation from the roundtable: the highest adoption the authors had seen was around 60%. It was unclear whether this was a ceiling; he attributes the obstacles primarily to resistance from organisations and people rather than technological limits.

Another point from the same table: the companies that gain most from agents are the ones that chose boring technology. If the stack is Python, TypeScript and JavaScript, or Java with Spring, there is an enormous supply of ready examples the models were trained on, so the assistant generates boilerplate well, performs refactorings and helps make sense of a project's structure.

On more exotic languages the picture is thinner: at Flo Health the main stack is Scala, plus Python and JavaScript, and in Copilot adoption Python comes first, then JavaScript, and very little Scala. Alexander confirms the same split: the people unhappy with the models are usually those writing in rare languages, where quality is noticeably worse — and he is tempted to reply: write in a technology once it is popular and simple enough, and everything flies.

The first maturity stage is adoption and utilisation: who uses the tool and how intensively. You count DAU, WAU and MAU, how many people use suggestions in the IDE, in how many pull requests the tool took part, how it showed itself in code review.

Alexander gives an example from an internal podcast recorded the day before: an SRE assistant that gathers incident information and writes the timeline for the post-analysis. The metric there was not "liked it or not" but the share of generated text that survived into the final report: around 50% in the case described. The agent produces a draft and a list of events; a human adds the narrative and any missing details.

At Flo Health they add a tag on the pull request saying it was made with an agent, the number of comments and edits afterwards, and post-production metrics: did that merge cause an incident.

Next comes impact — now in terms of the DX Core 4 metrics — and only then the cost-benefit question: what do I get, what does it cost, does it add up. The temptation to skip ahead is strong: utilisation looks too blunt, costs are easy to count, and impact is guessed. The trap is that irrelevant or dishonest metrics inflate expectations and cost trust; it is more honest to say the team chases utilisation now, impact next, and will show the money effect later.

Estimating impact itself needs telemetry, clearly defined engineering tasks and correctly worded survey questions — the difficulty jumps immediately. So use cases get prioritised: each needs a hypothesis and a clear metric, even a proxy one. If utilisation cannot be defined for a use case, Evgeny suggests taking the next, more tractable one — the feedback loop stays short.

One separate story is the time axis on the diagram: Evgeny liked that a direction of travel was drawn, and Alexander admitted he had added the arrow himself, because the original table did not read as a set of stages; the reply compared it to the scale in Team Topologies and suggested offering the addition to the authors.

On the rollout, Evgeny speaks first about psychological safety: this is not a KPI for judging individual performance but organisational lead indicators, even though metrics such as pull requests per engineer still pull towards the personal level and, unexplained, produce pushback and gaming. Then the purpose: an assistant augments the engineer inside current processes rather than inventing new ones.

To illustrate market comparisons, Alexander recalls the Hogan test, which returns percentiles rather than answers, while Evgeny cites Jellyfish with per-team percentiles, where 80% sprint predictability is already high against the market.

DX itself, Alexander notes, has outgrown its beginnings: it started with surveys and then built adapters to systems, computing metrics from code, tasks and releases; surveys remain useful for a quick start and for perceived quality or developer experience, while activity data reveals how the processes work. Evgeny adds that their questions are worded so carefully they can go into engagement surveys as they are. The next episode is promised on DX Core 4 with its Speed, effectiveness, Quality and Impact.