The right fit: model, harness and task
Benchmarks break, and model rankings depend on the harness: what it means for agent evaluation
Slide contents
1. The right fit: model, harness and task
Benchmarks break, and model rankings depend on the harness: what it means for agent evaluation
2. Swap the harness, the models swap places
Terminal‑Bench 4, 63 tasks: Claude Opus 5 and GPT‑6 Astra in OpenHands and in PI
3. #45 added effects; here they interact
Harbor: score = model + harness; here every model meets every harness
4. SWE‑bench Verified stopped measuring the frontier
138 hard tasks, at least 6 engineers each: 59.4% have a defect
5. The replacement lasted under five months
SWE‑bench Pro: 23.3% → 80.3% in 8 months, about 30% of tasks broken
6. Without answer leaks, GLM‑5.2 lost 21 points
SWE‑Bench Pro Verified v3: protection on all 731 tasks, 102 repaired
7. Nine lines of conftest.py score 100%
BenchJack: 9 of 10 popular benchmarks are hacked almost to the ceiling
8. Fixing the benchmark reordered the harnesses
TB 2.1 fixed 28 of 89 tasks: Opus 4.6 in Claude Code +12.1, in Terminus 2 +0.9
9. 66 configurations, one counted run per task
4 harnesses × 5 models × 3 task sets plus Codex–GPT and Claude Code–Claude
10. One TB4 task is 1.6 points
How to read the numbers: mean reward, fixed denominator, no overall score
11. Model order depends on the harness
TB4: Claude − GPT = +7.94 / −17.46 / −30.16 / −12.70 points
12. The native harness is not always best
Claude Code leads on two task sets of three, Codex on none
13. The best harness changes with the task
4 of 5 models change their winner across TUA, ALE and TB4
14. Kimi wins in openJiuwen on every set
+5.61 / +6.91 / +11.11 points, but −2.32 on ALE computing tasks
15. Pricier does not mean better
GPT: 60.32% in PI at $4.66 per task, 52.38% in DeepSeek Harness at $19.94
16. Models repair; harnesses return the signal
10 TB4 pairs, OpenHands vs PI: the model started 180 of 192 responses
17. A timeout turns hangs into hints
One TUA task, one model, Kimi K3: openJiuwen scores 1 in 11 minutes, PI scores 0
18. The agent signs off its own work
35 of 35 reports on failed tasks claim that everything was verified
19. Reward alone poorly filters training data
CAPTCHA: the best reward goes to the trajectory that bypassed a bot check
20. One run cannot measure variance
Variance unmeasured, settings unmatched; TB 4.1 then regraded the tasks
21. Neighbours measure the harness effect differently
Two studies find no robust score gap; Harbor finds only a small one
22. Evaluate the triple, not the model
Benchmark and harness versions, resources, runs and an explicit criterion