Skip to content
Primary sourcePodcastHarness atlasAI-native measurementCode became cheapPrevious reviewNext review
Research Insights Made Simple
Software factories · series 4/8Research Insights Made Simple

Tests pass, but don’t merge: factory code quality

Why accepting T2 should look at the next task, T4

/ Research Insights Made Simple · Software factories 4/8 · Code quality

Slide contents

  1. 1. Tests pass, but don’t merge: factory code quality

    Why accepting T2 should look at the next task, T4

  2. 2. Two passing patches, different futures

    The next agent stumbles more often on agent code

  3. 3. T2 acceptance does not yet test T4

    Bridge from the previous review: correctness is not evolvability

  4. 4. Four questions about code after acceptance

    Would it merge, can it grow, why not, how to accept

  5. 5. About half of passing patches wouldn’t merge

    The grader passes 24.2 points more than maintainers

  6. 6. Rejections hit patches that solve the issue

    Code quality and breaking other code are separate reasons

  7. 7. Cursor’s speedup lasts two months

    Lines +281% in month one, then no significant effect

  8. 8. Warnings and complexity rise and stay

    Warnings +30%, complexity +42%, and they stay

  9. 9. Swap the first author, keep the task

    PR0 → PR1 → PR2; compare AA with HA

  10. 10. Both first patches pass the old tests

    Pass-to-Pass green, Fail-to-Pass still red

  11. 11. One run: small gaps may be noise

    1,377 tasks in the set, fewer compared; one run per pair

  12. 12. Agent code usually hurts the follow-on task

    Up to −13.1 points; FeatBench has only 16 tasks

  13. 13. Refactoring suffers the most

    Our view: T4 resembles a feature with refactoring

  14. 14. Complexity metrics do not explain the gap

    PR1 complexity metrics: odds ratios near 1

  15. 15. Error-contract drift is linked to failure

    IEC: odds ratio 1.83; the model explains little

  16. 16. A silent fallback instead of an error

    PR1 tests are green, but the contract changed

  17. 17. The drift survives the next patch

    It stays in 85.9% of cases, rewritten in 6.9%

  18. 18. Agents differ more than agents and humans

    90-day reverts: 6.1% to 14.5% versus 11.5% for humans

  19. 19. Accept T2 by the outcome of T4

    Own code, same agent, a reference T2 alongside

  20. 20. Telemetry must see the next change

    DX PR size and confidence are symptoms, not the mechanism

  21. 21. Two steps are not years of maintenance

    Short chains, a quasi-experiment, three repositories, non-random tasks

  22. 22. Accept code by the next change

    Tests are the floor; evolvability needs its own check