Justin Reock: code is easier to change, harder to release confidently (#AI4SDLC)
Justin Reock’s new talk at DX continues the theme of his earlier talk, about how the processes and tools surrounding engineers affect the results of adopting AI. In “The State of AI in Software Development: Data from 400+ Orgs,” DX’s Deputy CTO returns to that subject. His argument is that faster code generation is constrained by the way the rest of development works.
The data Reock presents contains an interesting divergence:
- Developers’ assessment of code maintainability rose by almost 4%;
- Confidence in changes fell by 6%;
- Average PR size rose from about 44 to 72 lines.
These are DX’s observations; the first two measures are engineers’ own assessments. They do not establish that AI caused all of these changes.
The talk’s central point is that code has become easier to change, but releasing it has not become less worrying. What interests me is the gap between understanding something locally and trusting the behavior of the whole system. An agent can explain a function and help rewrite it. Questions remain: which contracts did we affect, what do the tests actually check, and which dependencies never made it into the context? An elegant explanation of a diff does not answer them on its own.
Reock gives an example: AI generates functions instantly, while the build still takes 45–60 minutes. That creates a temptation to bundle changes into one large PR. This is a hypothesis about a mechanism, rather than an established explanation for PR growth across the entire sample.
I would check how much work we pass to the next person. The author finished their part faster, but the reviewer now has to reconstruct the logic of several changes at once. If verification uncovers a problem, how long will rework take? Will any benefit remain afterward? Without those questions, it is easy to mistake local time savings for an improvement in the whole process.
This is a good reason to return to DX Core 4, which I have discussed before. It looks at productivity through speed, effectiveness, quality, and impact. Here I would look at PR lead time alongside size, returns for rework, and failures after release. A confidence survey matters too: it helps identify areas where the team is afraid to change code, even as the count of delivered changes grows.
The technical side also appeared in my discussion of Max Kanat-Alexander on DevEx for agents: fast, predictable tests, clear errors, documentation with context, and a manageable review workload. If an agent runs tests frequently, every extra minute of waiting repeats with that cycle. That makes a very concrete case for finally addressing slow CI :)
Still, I would not turn PR size into a standalone KPI. You can split one coherent change into ten awkward pieces and produce a lovely chart. What matters is whether the reviewer can understand the intention, check the consequences, and roll the change back if necessary.
So after adopting AI, I would ask the team: are we bringing changes to a state we are willing to stand behind faster, or are we merely filling the review queue faster?
P.S. The thumbnail naturally puts the question more provocatively :)
#AI4SDLC #Engineering #DevEx #Productivity #Metrics #Management