SWE-rebench: Lessons from Evaluating Coding Agents — Ibragim Badertdinov, Nebius (Category AI4SDLC)

I watched a short one. report Ibragim Badertdinov Nebius SWE-rebenchHow to honestly understand that the coding agent really knows how to solve software engineering problems, and not just beautifully passes the familiar benchmark.

The main problem with old and static benchmarks is that they quickly cease to be fresh. Tasks are publicly available, get into training data, prompts, scaffolding, retry cycles and non-obvious fitting appear around them. As a result, the resolved rate can reflect not only the ability of the model to understand the code, but also contamination, engineering bandage around the model or a successful launch.

SWE-rebench is trying to fix that. The idea is to regularly collect fresh GitHub issues and evaluate agents in a more production-like mode. The agent needs not to answer the question, but to go through a small development cycle: understand the issue, understand the repository, reproduce the problem, change the code, run checks and give the patch. The report clearly shows how complex the assessment infrastructure itself is. A good benchmark task should not be too vague, too obvious, too fragile, or tied to unstable tests. Otherwise, we begin to measure the noise: flaky environment, strange assert in the test, unsaid issue or random luck of the agent. That is, a good eval for agents becomes like a small development booth, rather than a set of tasks from the Olympiad.

The most important part is reward hacking. Badertdinov disassembles cases where a strong agent does not “write bad code” but uses available channels too well. First, he finds a future edit through git log. After cleaning up history goes to read the source issue and PR on GitHub. When web access is restricted, it tries to access the same information through curl. The lesson here is not that the model "cheats" in the human sense. The lesson is that an agent system has a much wider reward hacking surface. If the agent has a terminal, network, repository history, and tools, then you need to evaluate not only the final diff, but also the trajectory: where did he get the information, which commands he ran, whether he used a future response.

Another important shift is the economy. For production agents, it is not enough to know the average resolved rate. Tokens per problem, cost per problem, cached tokens, number of attempts, SEM pass@5 stability between several launches. One successful run can be luck. The five launches already show how reliable the model is, rather than just occasionally hitting the target.

SWE-rebench is not just a leaderboard. On HuggingFace. version V1 contains a description of 21,000+ interactive Python SWE tasks for RL training agents, and for version V2 already indicated 32,000+ environments in 20 languages 126,000+ additional tasks. That is, the same pipeline becomes both an assessment tool and a training environments factory.

In general, if we want to run coding agents in real development, we need to measure not only “decided or not decided”. It is necessary to measure the freshness of tasks, leaks, cost, reproducibility, reliability, environmental limitations and the behavior of the agent by steps. And you can start with public benchmarks, and then move on to measurements on internal tasks.

#AI #AI4SDLC #Engineering #Agents #Evals #DevTools #Software