[1/2] SWE-PRBench: Benchmarking AI Code Review Quality Against Pull Request Feedback (Category Whitepaper)
When figuring out how to evaluate the quality of agents inside the SDLC came across an interesting and fresh job Deepak Kumar, an independent researcher. SWE-Bench-like benchmarks measure whether a model can write the right patch, rather than whether it can rate someone else’s PR as a reviewer. And in the proposed approach, the author does exactly the second: takes real merged PR and real comments of people in the review as ground truth, does not synthesize comments and does not reduce the task to the similarity of the text. In addition, the authors capture three modes of context. (progressively)To check the effect of its expansion
- config_A (diff + summary)
- config_B (File context added)
- config_C (and test/context layers.) This approach allows you to measure how good different models are at finding issues compared to people.
If you delve deeper into the methodology, it is 1) First they select repositories. RQS (Repository Quality Score)Review culture, freshness of PR, quality of tests/CI, volume of PR activity and contamination proxy. Repositories below 60/100 discarded. The strongest signal is the culture of the review: the share of meaningful comments from people in recent merged PR. 2) Then there's the PR. 10- step filtration: only sturdy PR, at least two meaningful human comments, there are changes beyond tests, not only changes to documentation, not just an automatic dependency bump, not only an AI review, diff can be parsed, the basic commit is available, the repository is still public, and the RVS threshold passes (What was mentioned in the first paragraph). about 3,000 Raw PR remains in place. 700 candidates, after RVS is trimmed to 350 PR 65 repositories. Ground truth is taken from real-world reviews via the GitHub API; comments are not generated or edited manually. Authors separately filter AI/bot comments and exclude PR, where there are more AI comments 30%. 3) An important part of the methodology is built around taxonomy Type1 - the problem is visible directly in diff Type2 – Understand the surrounding code in the same file that has not changed Type3 – Reflection on cross-file dependencies This move turns “code review quality” from one murky metric to three different cognitive modes. 4) The evaluation is also interesting. The agent must return 4–6 issues in JSON with severity, and the judge model classifies each comment as CONFIRMED, PLAUSIBLE or FABRICATED. PLAUSIBLE is a good approach, as the authors admit that people’s reviews are not exhaustive, so a valid new comment is not considered an automatic hallucination. Next comes the mapping so that the model does not receive double points for breaking one idea into five comments, and the final score mixes completeness, accuracy, semantic alignment, actionability and efficiency, penalizing hallucinations and redundancy.
As a result, interesting results were obtained:
- On them. 100PR stratified sample no model catches more 31% human-flagged issues; average detection rate 8 Config/A models are approximately 26%. Top-4 The models show roughly the same aggregate score.
- On the profile of models, we can see that there is no better reviewer and we have the usual precision/recall trade-off. DeepSeek V3 gives the best raw detection on config A (DR=0.312)GPT-4o is the best hallucination profile. (FPR=0.193). Llama. 3.3 70B worst FPR -- 0.417.
- All 8 Models degrade from config\ A to config\ C. Config and config are different. 500 token (2,000 vs 2,500) The problem seems to be how the context is presented, not just the volume.
- Things get worse on the transition from config\ A to config\ B: Sonnet's Type2 score drops 0.22 before 0.10 in the transition A→B; in DeepSeek - with 0.20 before 0.10. That is, models break down not where you need the entire repository, but on the step of adding the surrounding context from the modified file.
In continuation Let’s talk about what all this means in practice.
#AI #Engineering #Software #SystemDesign #Agents