Skip to content
back to the archive page
#AI4SDLC

How incident.io measures AI spending (Series #AI4SDLC)

At incident.io they calculated AI spending two ways: per engineer per month and per effort point, a unit of complexity for shipped changes. A PR of average complexity counts as one point. A model estimates the weight of the rest.

Gergely Orosz shares incident.io’s figures in his newsletter and discloses that he’s an investor. Switching from older Anthropic models to Opus 5.5 brought spending down from about $4,500 to $1,600 per engineer per month. On cost per effort point, the infrastructure team stood out. Its figure was about five times higher than almost everyone else’s. Engineers use an agent to understand production behavior, run analyses or benchmark a system. That costs money even when there’s no PR at the end.

The company has figures on what happens after code generation, too. The share of PRs reaching production without human review rose from 12% to 43%. Incidents per thousand PRs fell by 60%, while the absolute number of incidents rose. They ship far more changes now. The team’s analysis suggests incidents increasingly happen because people lacked information about production behavior when preparing a change. They’re trying to feed operational signals back to the agents.

In my long read on AI economics, I suggested counting the cost of an accepted task, including verification, rework and the cost of a mistake. The infrastructure example shows why it’s worth agreeing on what counts as a result. If an engineer and an agent found and verified the cause of an outage, they did the work even if nothing in the repository changed.

The incident figures pick up a thread from my talk on AI4SDLC bottlenecks. As code gets written faster and review holds up fewer releases, more depends on what we know about the system after release. Fewer failures per PR and more failures per month can easily coexist on a dashboard. To discuss ROI, I’d also want to know how severe those failures were and how much on-call time they took.

One more detail that reminded me of “The Junior Engineer After Coding”. New hires at incident.io, including experienced engineers, are encouraged to spend their first month without AI to learn the codebase. I find the choice interesting in itself: you can spend that month understanding the system, even if an agent could help you land your first PR sooner.

#AI4SDLC #Engineering #Metrics #FinOps #SRE