SWE-bench: How to Choose an Agent When the Leaderboard Cannot Distinguish the Leaders (Series #AI4SDLC)

One agent scores 79.2%, another 78.8%. Should we buy the winner? Across 500 tasks, the gap is just two additional solutions. What interests me here is the point where precise numbers start promising more than we have actually measured.

In "Coding Agents Have Converged," Fengshuo Liu of Imperial College London and coauthors examined 254 published SWE-bench results. The paper came out on September 15, 2026 (version v1 of the paper), and the authors released their analysis code. They did not run new agents: they examined what conclusions the already collected data from agents in 2023–2025 could support. SWE-bench Verified consists of 500 GitHub tasks: an agent prepares a patch, and an evaluation harness runs the tests. The top ten in the studied sample share 285 successes and 51 failures. Their outcomes differ on only 164 tasks.

Imagine two candidates taking the same test. Each solves 396 problems: 378 in common, plus 18 unique to each. Those are the actual numbers for the two leading entries. Their overall scores are identical, yet they may help with different individual tasks.

To compare the candidates, look at their wins against each other. On tasks solved by only one of them, who wins more often? Is the advantage convincing enough? That is roughly what the paired McNemar test used by the authors checks. None of the 29 adjacent pairs in the top thirty showed a statistically significant difference. That does not make all thirty identical: differences can be detected between some entries further apart. But the rank numbers create an overly confident picture that is not particularly well supported.

The agent's scaffold also matters: how it retrieves context, calls tools, checks its work, and retries. For one model, the observed range across scaffolds reached 29.8 percentage points. However, teams, versions, and run conditions differed. This is a reason to evaluate the complete combination; promising that improvement from a scaffold swap would go too far. I would also read the headline about agents having converged cautiously. The study covers results from 2023–2025, one run per published result, and one benchmark family. There was insufficient evidence of superiority for adjacent entries. That did not establish equal capabilities.

There is room to disagree with the authors. They describe complementarity as limited: the two leaders together cover 414 tasks rather than 396. But those extra 18 amount to roughly 17% of the first agent's failures! That could matter for expensive tasks. Of course, we need a way to select the correct patch and compare a second agent with another attempt by the first. Taking the union of their answers does not settle that.

How would I choose? I would use the following procedure, which is not described in the paper itself but looks practical

- Pick 2–3 candidates that fit your access requirements, integrations, and budget. Record the models, scaffold versions, time limits, and attempt limits. - Give them identical real tasks: bugs, API changes, tests, and work in unfamiliar code. Start with a few dozen for a pilot. That can reveal large differences, but it does not promise to establish an advantage of a couple of percentage points. - Check patches with your own tests and review. Repeat at least some runs: one successful attempt does not establish reliability. - Measure the cost of an accepted solution: model calls, waiting time, and human minutes spent reviewing and reworking it. Track regressions separately. Break results down by task type; an average can hide useful specialization.

If the quality advantage remains unclear, you can choose the cheaper, more convenient candidate while retaining that uncertainty in your assessment. Give the second a separate role: test how many of the first agent's failures it fixes for a comparable budget. That provides a testable reason to keep two agents in use.

#AI4SDLC #Agents #Research #Evals #Engineering