AIDev: Studying AI Coding Agents on GitHub (Category AI4SDLC)
Read it. short from 9 February 2026 Learn how to learn coding agents from real PR, not demos. It presents a public dataset on which agentic software engineering can be studied not by demos, but by the tracks of real pull requests. Authors are Hao Li, Haoxiang Zhang and Ahmed E. Hassan of Queen's University. This is a good sign of trust: Hassan has long been involved in empirical software engineering and mining software repositories. Paper submitted to arXiv 9 February 2026 year and associated with MSR 2026 Mining Challenge; data is posted on Hugging Face and Zenodo, code and laptops on GitHub.
AIDev dataset aggregate 932 791 pull request, which the authors refer to Agentic-PR: PR, created by coding agents. There are five tools in the sample: OpenAI Codex, Devin, GitHub Copilot, Cursor and Claude Code. These PRs are distributed across 116 211 repositories and associated with 72 189 developers; cutoff - 1 August 2025 years. For deep analysis, there is a subset: 33 596 PR 2 807 repositories with more than 100 stars. There are already review comments, review verdicts, commit-level diffs, related issues, timeline events, and automatic task type classification by Conventional Commits.
Methodologically, this is a dataset paper with the main result in the form of infrastructure for interesting questions: Who's using agents? (Juns, Midles, lords) What kind of PR are reviewed and merge Does the description of PR match with diff?
- Do agents raid tests? What security/quality patterns pop up in agent-authored code
For me, this is the most interesting shift in the unit of measurement. AI4SDLC has long measured either completion in an IDE or synthetic benchmark like “solution issue.” AIDev offers a look at the whole lifecycle: issue -» PR -» commits -» review -» comments -» CI/merge/close -» дальнейшая судьба изменения. It's much closer to a real engineering system. The practical application here is quite direct. If a company implements coding agents, it should build a similar internal telemetry. Not “how many lines are generated” and not “how many licenses are activated”, but what happens with agent-authored PR: diff size, share of tests, time to review, number of comments, merge rate, rollback/hotfix, security findings, conventions and CI compliance.
Separately, I note that the study has several caveats: The dataset is built on public GitHub, so enterprise development and closed repositories almost certainly look different. The quality of the conclusions depends on how well the agent-authored PR for each tool is defined; in the short version of the paper, this layer is described in insufficient detail. (I'll have to dig deep.) A subset of PR from repositories with 100+ stars shift the picture towards noticeable open-source projects.
P.S. Next, I will tell you about some interesting articles that rely on insights that were extracted from this dataset.
#AI #AI4SDLC #Engineering #Research #Agents #Software #DevOps #Processes