AI made development faster: how can we tell?

In METR’s 2025 experiment, experienced developers worked on real tasks in familiar repositories. They expected to speed up and still thought they had saved time after doing the work. Measured task completion took 19% longer with AI on average. The 16 developers worked on 246 tasks in projects they knew, using tools available in early 2025. The 2026 update ran into fresh measurement problems, including participants reluctant to work without AI and the difficulty of accounting for agents working in parallel.

Microsoft observed about a quarter more merged PRs per engineer per day among early users of agents. The researchers checked for a similar increase using an earlier, placebo launch date and found none. Anthropic reported more commits per engineer and changes to work in other functions. Those results invite questions about what shipped and who benefited. Task duration, merged PRs and employee activity each answer a different question, so they cannot be read as three estimates of the same effect.

A team that can change the whole process can measure the path from choosing a problem and testing a hypothesis through release and evaluation. An activation team might track the share of new customers who take a meaningful first step within a week. It can pair that with elapsed time, the full cost of testing hypotheses including failed attempts, and a guardrail such as support contacts. This connects the pace of shipping with what happened for customers.

I recall teams moving feature descriptions from a wiki into their repositories. A developer could turn the brief into a specification, delegate implementation and take the change through release. Small teams with end-to-end responsibility and product engineers can also reduce handoffs. The number of handoffs, time spent waiting, incidents and team workload help establish whether the reorganisation worked. A process that succeeds in one team may be hard to transfer across an organisation where teams have different authority and appetite for change.

The first layer tracks who uses the tools, for which tasks and in what mode. Calculate active use among people with access and AI-assisted work among tasks that qualify for that kind of help. Working interactively with an assistant and delegating to an agent place different demands on an engineer’s time. Parallel runs can add up to hours of agent time even when the engineer spends only a few minutes assigning the work and returns later to check it.

The second layer ties time to the work engineers are trying to accomplish. In the Google approach I discuss, events from tools are grouped into sessions, linked to artefacts and mapped to work such as understanding existing code or reviewing a change. Telemetry that begins at the first commit misses the preceding work. Flow metrics can show where queues grow after coding gets faster. In my experience, changes piled up at review and approvals, and surveys and interviews helped explain the delay.

The third layer weighs the gain against model usage, infrastructure, platform support and human verification. Freed engineering hours have economic value when the team can identify what those hours made possible. Shipping more features can also yield diminishing returns as the team works through lower-value items in the same backlog.

For AI code review, compare the current bot rules with a revised set. Across comparable batches of PRs, measure the time spent triaging comments and the cost of running the bot. Check whether its findings were actionable and which defects made it through. The author addresses findings and the human reviewer evaluates the remaining risk. Those observations inform where the bot helps and which checks the team still needs.

The slides illustrate the calculation with a batch of 100 PRs. At an hourly rate of $60, eight hours of work and $30 in AI usage cost $510. Five hours and $20 in AI usage under the new rules cost $320. The $190 difference estimates resources freed in this example, assuming comparable review quality.

For a migration, compare the estimated manual effort with the cost of preparing rules, running agents, reviewing output and handling exceptions. The number of similar changes helps determine when that setup pays off. The last few exceptions can keep the old system running, and some regressions surface only after release. Include the work required to retire the old system.

For a prototype, the outcome is a tested hypothesis. A product team may assemble an interface sooner, while a safe rollout to users still takes platform work. A successful experiment may then need to become a production feature. Count those costs alongside the time and effort of running the experiment itself.

In the Q&A, I suggest grouping varied work into categories such as interface changes and partner integrations. The same team may see different effects across the two categories. If integration work stays slow, check whether agents can access rules that engineers have kept in their heads.

On quality, I draw on DORA’s observation that AI amplifies existing practices. A team used to maintaining tests and automated architecture checks may find those practices cheaper. A team without those habits may ship more changes and then spend its time fixing defects. Measure the quality of the resulting work alongside speed.