Model or harness: what moves the score
8 models and 4 harnesses on 54 benchmarks: which moves the score more, and at what cost
Slide contents
1. Model or harness: what moves the score
8 models and 4 harnesses on 54 benchmarks: which moves the score more, and at what cost
2. Four stops: from adapters to the index
Infrastructure, model or harness, cost and failures, Harbor-Index
3. #38 swapped parts; Harbor swaps harnesses
#38: one loop, four models; Harbor: 8 models, 4 harnesses, 54 benchmarks
4. Adapters turn m × n into m + n
One adapter per benchmark and one integration per agent instead of one per pair
5. A Harbor task has four parts
Instruction, environment, tests and a reference solution: the Terminal-Bench schema
6. Each adapter is checked against the original
Same agent and model on both sides, k runs, mean ± standard error
7. Every model runs in two harnesses
Its vendor's native harness and the neutral Terminus-2: 16 configurations on 54 benchmarks
8. First, subtract benchmark difficulty
Equation 7: score = model + harness + benchmark shift + noise
9. Model spread is 5.2× harness spread
From GPT-5 nano to Gemini 3.1 Pro is 0.451; across harnesses, 0.087
10. Native harnesses add 0.5–9 points
Our calculation: native minus Terminus-2 — Claude +0.042, GPT +0.087, Gemini +0.005
11. Harnesses are compared only through Terminus-2
Each native harness saw only its own vendor's models
12. Frontier models pay off on medium tasks
Frontier gain over the five other models, pp
13. Fewer tokens, yet a pricier trial
Frontier models save the most tokens on easy tasks
14. Wrong answers lead in every harness
GPT-5.4 hits environment blocks nearly 3× as often in Terminus-2
15. 82 of 6,627 tasks made the index
Difficulty filter, AI auditor, human reviewers and an audit-and-fix loop
16. A third of “hard” tasks were broken
A zero score often signals a task defect, not a model limit
17. Native harnesses help the top three
Harbor-Index 1.0: pass rate, native harness versus Terminus-2
18. A foreign harness costs several times more
Via OpenRouter, Claude Code costs 3.7–12.3× Terminus-2 per run
19. Timeouts aren't always the model's fault
Matched task by task, native harnesses cut the timeout rate from 42% to 26%
20. The gap depends on what you compare
Seven harnesses on three models; three on two
21. The blog and v3 report different numbers
The GigaConf slide cited the blog: “no comparison is significant”
22. The findings don't transfer to every model
The authors' limitations and the host's caveats on the paper's numbers
23. Model first, then harness and cost
What to take from the paper into your own agent evaluation