Comparing AI Coding Agents: A Task-Stratified Analysis of Pull Request Acceptance (Category AI4SDLC)
I read this one. paperIt shows why ranking agents without considering the type of task are not very useful. This article shows that the correct answer to the usual question "what coding agent is the best?" is it depends:) For their analysis, they took a PR kit from GitHub called AIDev, which I'm talking about. told Recently. And then they analyzed the merge rate, but not just the general rate, but given that one agent can more often receive documentary tasks, another - bugfixes and features. As a result, they can not be compared as if they received the same load.
The work was done by Giovanni Pinna, Jingzhi Gong, David Williams and Federica Sarro from the University of Trieste, King's College London and University College London. The article itself was accepted by MSR'26 The Mining Challenge Track is essentially an observational study, not a randomization experiment, so here we can see correlations rather than causation.
The methodology looked like this. 1же The authors took the substrate of AIDev-POP, i.e. AI-generated pull request's from GitHub repositories 100+ stars. From 33 596 PR left out 7 156: Closed PR only, MIT/Apache-2.0 repositories, and only those PR, where before closing there was at least one review or comment not from the author. Success was defined as acceptance rate: PR was merged or not merged. 2Then the most useful thing began. The authors did not just compare OpenAI Codex, GitHub Copilot, Devin, Cursor, and Claude Code with one table. They decomposed PR by task type: docs, feat, fix, test, refactor, chore, build, ci, perf, etc. Then we looked separately at weekly dynamics, task distribution and pairwise comparisons within task types.
What are the results? 1Global numbers look seductive: OpenAI Codex - 77.9% acceptance, Cursor - 74.5%, Claude Code - 71.9%, GitHub Copilot - 68.0%, Devin - 61.6%. Without context, this table cannot be read correctly. Claude Code has a sample of everything. 139 PR, Devin - 2 252Copilot's - 2 194. The loads are also different: Copilot sits more often in fix tasks, Claude Code sits more often in features, Cursor has a shorter concentrated observation period. 2The main result for me is not the winner, but the scale of the interfering factor. (confounding variable). The type of task makes the difference stronger than many differences between agents. Chore-tasks in their sample are taken in 84.0% of cases, perf - in 55.4%; between categories is obtained 29 p.p. Among the mass tasks docs give 82.1% and features-- 66.1%. That is, an agent who gets more docs may look “smarter” simply because it has easier workload. 3️ According to the time dynamics, Devin is most noticeable: the authors see it as the only stable positive trend.0.77Percent acceptance rate per week 32 active weeks, approximately from 60percentage 80%. But they cautiously write that causality does not follow from this: it can be improving the model, training users, changing the types of tasks, concentration in other repositories or all together. 4By task-stratified comparison, the picture becomes less convenient for the leaderboard. Codex is consistently strong in categories: 59.6percentage 88.6%, especially fix and refactor. Claude Code leads in docs and features, but some of these cells are small, so generalizing is risky. Cursor looks good in fix/test scenarios. Significant differences after the Bonferroni correction (multi-comparison) Mostly they pop up in fix and feat, not in all tasks in a row.
The limitations should be discussed separately. Acceptance rate is not code quality. Merged PR can bring a bug, tech debt, or security issue. Public GitHub is not equal to enterprise development. Repositories with different review cultures can accept or reject PR by their own rules. Repository-level clustering is not modeled in Chi-square tests. Task labels come from the AIDev pipeline, not from the manual markup of every PR. Samples by agent are asymmetrical in time and size. But that's why paper is useful. It teaches not to choose an agent according to the overall beautiful number, but to build implementation metrics correctly.
If a team implements coding agents, I would take away the rule of thumb: any internal analytics should be broken down by task. (task-stratified). Compare not agent A vs agent B, but docs to docs, bugfix to bugfix, tests to tests, feature to feature. Separately, see review load, number of comments, size diff, CI, rollback, static analysis, found defects and support after merge.
#AI #AI4SDLC #Engineering #Research #Agents #Software #Evals