Live · 21 October 2026 · 10:00 MSKResearch Insights Made Simple #42
Agent evals: telling whether an agent succeeded
What counts as success, who checks it, and when a low score means a broken test
/ Research Insights Made Simple #42 · 21 October 2026 · 10:00 MSK
Slide contents
1. Agent evals: telling whether an agent succeeded
What counts as success, who checks it, and when a low score means a broken test
2. Claude Code: feedback first, then evals
3. A “done” reply is not the outcome
4. We evaluate the model with its harness
5. Suite, task and trial are different denominators
6. The trajectory explains; the outcome confirms
7. Code, models and humans check different things
8. Scoring rules change what scores mean
9. Fix failing tests without breaking passing ones
10. Conversations are graded on three dimensions
11. A citation does not prove a claim
12. A confirmation page is not an order
13. One successful run can hide instability
14. Track growth and regressions in separate suites
15. Start with 20–50 tasks from real failures
16. Test both action and restraint
17. Every trial starts from a clean environment
18. One judge per dimension, checked against experts
19. Low scores can come from broken tests
20. Read trajectories before trusting the score
21. No single layer catches every failure
22. Evals turn “feels worse” into evidence
23. Petri checks the realism of evaluations
24. Autonomy also needs measurement after deployment
25. Evaluations missed real Claude Code regressions