Research Insights #44 Materials: Randomness in Agent Evals (Series #AI4SDLC)

My solo Research Insights Made Simple #44 episode came out on November 2, 2026. Suppose an instruction change takes your agent from solving 31% of tasks to 33%. Do you ship it? In “On Randomness in Agentic Evals”, the same DeepSWE-preview and R2E-Gym combination ran on SWE-bench Verified ten times. The share of tasks solved ranged from 31.6% to 37.0%. I used that example to discuss how to evaluate small improvements.

In the episode

🔸 Why variation remains at zero temperature Even with greedy token selection, precision, parallelism, and the way an engine batches requests can affect the computation.

🔸 How to read the table of runs Columns show how many tasks the agent solved in each run. A row shows how many times it solved a specific task. Identical overall scores can hide different failures.

🔸 When pass@k and pass^k are useful If tests can verify a patch quickly and you can afford multiple attempts, you may want to know whether any attempt works. If the job needs to be done reliably, you want to know how often all attempts work.

🔸 Where trajectories diverge. On a Django task, two runs with Qwen3-32B at zero temperature diverged at token 94. One agent started by searching a specific file, the other searched the whole directory. Only the second patch worked.

🔸 How many times to run a benchmark A/B testing gives you a way to think about this. Decide in advance what gain you want to detect and estimate the number of repetitions given the variation. The study measured it on one benchmark with open models and no history compaction, so you’ll need to measure your own harness.

Episode materials 📌 Episode page with timestamps and slides 🎬 Video: YouTube, VK Video 🎧 Audio: Podster, Apple Podcasts 📖 Written summary

If you’ve compared two versions of an agent, what improvement would justify switching models or harnesses for you, and how many runs would you spend to find out?

#ResearchInsights #AI4SDLC #Agents #Evals #Engineering