On Randomness in Agentic Evals: Noise in Agent Evaluations

In “On Randomness in Agentic Evals,” researchers at KTH tested how much an agent's evaluation changes when you repeat the same experiment. I especially liked that they measured the variation at temperature=0 separately. It was still there. This was a sizeable experiment: 500 SWE-bench Verified tasks, three models, two agent harnesses, two temperature settings, and ten runs per configuration. That came to 60,000 trajectories, 25.58 billion tokens, and 1.88 million tool calls.

The difference between the worst and best of ten runs ranged from 2.2 to 6 percentage points, depending on the configuration. For example, Devstral-2 with nano-agent at zero temperature solved between 60.6% and 66.6% of the tasks. The agent itself hadn't improved between runs. A gain of a couple of points on a single run is therefore insufficient evidence of progress.

Randomness also helps find solutions. For DeepSWE-preview with R2E-Gym at temperature=1, the authors found:
– pass@1 of 34.4%: the probability of success on one attempt.
– pass@5 of 52.9%: at least one success in five attempts.
– pass^5 of 15.5%: success in all five.

All three estimates are averaged across tasks. The number in the metric name is how many attempts we're interested in. To estimate it, the authors used ten attempts per task. The difference between the last two metrics is the probability of getting both a success and a failure among the five. I find these interpretations more useful than one leaderboard number. They show the value of retries and how inconsistent results can be. You still have to be able to recognize a successful attempt.

The authors recommend estimating pass@1 from multiple attempts per task, reporting the spread, and working out the number of repeats required for the size of improvement you need to detect.

A few notes in the margins

1️⃣ I liked the choice of two harnesses: the authors' own nano-agent and R2E-Gym, which was used to train DeepSWE. Two models ran locally, while Devstral ran through the provider's API. That makes the observation less dependent on one implementation. It still doesn't let you isolate the effect of infrastructure, because the model changes along with it.

2️⃣ For context, the team chose the easiest setup to analyze. History only grows, with nothing cut or summarized. That gives a clear representation of the full trajectory, including tool responses. It leaves out how an agent behaves with context compression, though. A summary might retain different details or remove differences that don't matter. That's a question for the next experiment.

3️⃣ I wrote “butterfly effect” in the margin of the introduction, and the authors later use the analogy themselves. Trajectories diverge very early. For DeepSWE with nano-agent, for instance, the median first difference after the shared prompt is at token five at temperature=1 and token 56 at zero. A different phrase can lead to a different search, a different piece of code, and a different edit. The first textual difference doesn't necessarily indicate a different strategy, though. I'd like to measure the first divergence in actions alongside it and see when that changes the outcome.

4️⃣ The main obstacle to repeated runs is straightforward: they're expensive. The smaller the improvement you want to detect with confidence, the more computation you need. Before running that kind of experiment, it's worth agreeing on how big a gain would justify switching models or harnesses. Otherwise you can spend a lot of compute precisely measuring a difference that makes no difference to your work.

I'll discuss the paper in the Research Insights #44 live stream on Monday, November 2, at 10 a.m. Moscow time.

P.S.
We previously covered the September 17, 2026 whitepaper “An Empirical Study of Harness Design for Coding Agents” in Research Insights Made Simple #38: Designing a Harness for Coding Agents and What It Adds. It used one run per task at zero temperature, without estimating the spread across runs. It did compare 176 configurations for planning, tools, and context management. I wonder how the results would have looked if the authors had used these approaches to reduce noise :)

#AI4SDLC #AI #Agents #Research #Evals #Engineering