Randomness in agent evals: improvement or noise?

The earlier episode on agent evaluation explained how to check an outcome. This review asks why the resulting number deserves confidence. Alexander compares the problem with laboratory measurements and product experiments: measuring an effect is only useful when its uncertainty is understood. In the paper’s hypothetical example, an old version solves 31% of tasks and a new version solves 33%. With one run per version, the difference cannot confidently be attributed to a prompt or harness change.

A real result illustrates the concern. Unchanged DeepSWE-preview in R2E-Gym at temperature 1.0 scores between 31.6% and 37.0% across ten runs, averaging 34.4%. That is a 5.4 percentage point range. It explains why a two-point gain deserves further investigation. Falling inside an observed range is not itself a significance test; it shows why comparing two isolated scores is inadequate.

The authors use SWE-bench Verified, containing 500 repository bug-fixing tasks whose outcomes are checked by tests. They compare Qwen3-32B, its agentic fine-tune DeepSWE-preview, and the larger Devstral-2. Each runs in two harnesses: the minimal nano-agent built for this experiment, and R2E-Gym. DeepSWE-preview was trained with R2E-Gym; a newly created harness lets the authors examine performance without that familiarity. They test recommended temperatures and zero, with ten repetitions of each configuration.

The experiment collects roughly 60,000 trajectories, consumes 25 billion tokens, and makes nearly two million tool calls. Both harnesses retain complete histories without compaction or summarisation. Alexander explains the measurements as a matrix: rows are tasks and columns are repeated runs. The success fraction in a column is one run’s score; averaging columns gives mean pass@1. Counting successes within a row describes consistency on a particular task. These are distinct levels of analysis.

Across the tested configurations, best-to-worst scores differ by roughly two to six percentage points. Setting temperature to zero neither removes this variation nor necessarily reduces it. Greedily choosing the next token does not make the whole system deterministic. Floating-point arithmetic, parallel execution, hardware-specific optimisations, and request batching can affect the computation.

The episode turns to Thinking Machines Lab’s explanation: numerical kernels may produce slightly different results at different batch sizes, while batch size changes with server load. A model setting alone therefore cannot guarantee reproducibility. Temperature can also change mean performance, which requires its own statistical assessment. Zero should not be assumed to improve an agent, and identical temperatures should not be assumed to produce identical answers.

pass@k measures success in at least one of k attempts. pass^k requires every one of those attempts to succeed. The former suits a pipeline with automatic verification and inexpensive retries. The latter matters when dependable execution is required and repeating or checking the work is costly. A single average score hides the distinction between available capability and reliable execution.

In an illustration, one task succeeds six times in ten attempts. Any selection of five observed attempts without replacement contains a success because there are only four failures. Its pass@5 estimate is therefore 100%. All five succeed in just six of 252 possible selections, about 2.4%. This calculation uses a finite sample, so simply raising the observed success fraction to a power gives a different answer. For DeepSWE-preview in R2E-Gym, pass@1 is around 34% while pass@5 reaches 52.9%: additional attempts reveal capabilities that a single run misses.

The authors retain system instructions, task descriptions, model responses, and command outputs, then locate the first differing token between two runs of the same task. At recommended temperatures, divergence can begin almost immediately. Zero temperature delays the split but does not prevent it. Alexander relates this to the butterfly effect: a small initial difference can redirect a long computation.

In a concrete Django task, Qwen3-32B with nano-agent at zero temperature diverges at token 94. One attempt ultimately succeeds and the other fails. Examining the trajectories makes the difference between final scores more understandable. It does not reduce every failure to one token or establish an identical mechanism across all tasks, but it demonstrates why the agent’s path should be recorded alongside its outcome.

The paper’s appendix calculates required run counts from an expected effect size, standard deviation, and statistical requirements. The analogy with A/B testing is direct: smaller effects need more observations. Here, repeated agent executions replace additional user traffic as the main experimental cost. Ten runs describe this particular experiment; they are not a universal recommendation.

A budget permitting only five repetitions may be insufficient to distinguish a modest improvement from noise. A statistically inconclusive result would then not establish that the change is useless. The practical sequence is to decide which effect matters, estimate variation, and calculate the required repetitions. Reports should include average pass@1, standard deviation, and scenario-appropriate pass@k and pass^k. This applies equally to selecting a cheaper model and evaluating an internal harness.

The study covers one benchmark of Python tasks, three open-weight models, and harnesses that do not compress their histories. It does not evaluate long, open-ended work or leading closed models. Variation is examined mainly across repeated runs; explaining why individual tasks differ in stability remains further work. Its numerical results should therefore not be transferred automatically to another agent stack.

Alexander also highlights unequal tool limits: nano-agent permits 500 calls, whereas R2E-Gym permits 100. An observed configuration advantage cannot be explained solely by harness complexity. The paper’s methodological value extends beyond its scores: it prepares the ground for later AGENTS.md and DeltaSelect discussions, where intervention effects must again be separated from randomness. Internal experiments should account for this uncertainty from the outset.