One run lies: noise in agent evals
60,000 SWE-bench Verified trajectories: how far one pass@1 swings and how many runs you need
Slide contents
1. One run lies: noise in agent evals
60,000 SWE-bench Verified trajectories: how far one pass@1 swings and how many runs you need
2. Four questions for one pass@1
Spread, two bounds, mechanism and run count
3. A two-point gain may be noise
One run per task is common practice for papers and leaderboards
4. 60,000 trajectories on one benchmark
Three models, two scaffolds, two temperatures, ten runs of 500 tasks
5. Runs are columns, tasks are rows
How to read what follows: a share per run, successes per task
6. Ten reruns spread over 2–6 points
Table 1: the worst and best of ten identical runs
7. Temperature 0 does not make runs identical
At zero the deviation is never below that at the recommended temperature
8. 11 points is real, 0.3 is noise
Temperature on nano-agent: DeepSWE-preview gains from randomness, Devstral-2 does not
9. pass@k and pass^k bracket the agent's score
Equation 3 on one task: 6 successes in 10 runs
10. Retries add 18.5 points; consistency costs 18.9
DeepSWE-preview on R2E-Gym; Devstral-2 on nano-agent sits tighter
11. Runs diverge within the first tokens
Medians on nano-agent; zero only delays the split
12. One token can flip a task's outcome
django__django-9296: Qwen3-32B, temperature 0, nano-agent
13. Two points take nine runs per version
Two-sample test: σ = 1.5 points, p < 0.05, 80% power
14. Half the effect, four times the runs
Table 2: runs per version at 80% power
15. AGENTS.md effects are as small as noise
Review 8 on KTH's noise scale: the host's view
16. 22 of 113 tasks clear the cut
DeltaSelect: 113 DeepSWE tasks; the 0.50 correlation cut is descriptive
17. Report mean, spread and both bounds
The authors' three practices; the talk's protocol adds a pilot and cost per success
18. Tested on one benchmark, without compaction
What the authors measured, assumed and left for later
19. One run is not yet a measurement
Ten reruns spread over 2–6 points
Temperature 0 does not remove the spread
pass@k and pass^k frame the agent's range
+2 points need 9 runs per version
Compaction and other benchmarks are untested
Count the runs before you celebrate a gain