Research Insights Made Simple · episode 44· recording pending
Randomness in Agent Evals: Improvement or Noise?
On Randomness in Agentic Evals
November 2, 2026
The release date has arrived; the recording is still being prepared.
Episode participants
A solo episode without invited guests.
What we will explore
Research Insights Made Simple episode 44 is scheduled for November 2, 2026 at 10:00 Moscow time. Alexander Polomodov reviews On Randomness in Agentic Evals and asks how to distinguish an agent improvement from random variation.
The authors repeat unchanged configurations on SWE-bench Verified, comparing three models and two harnesses. We discuss run-to-run variation, zero temperature, diverging trajectories, and the differences between pass@1, pass@k and pass^k.
We examine how to assess uncertainty and choose a repeat budget for your own comparison, considering cost and the study's limits: one benchmark, open-weight models and harnesses without context compaction.
AI in SDLCResearch methodology