
Same agent, different harness: what will runs show?
The harness series finale: the comparison design, first measurements and what a paid run needs
Slide contents
1. Same agent, different harness: what will runs show?
The harness series finale: the comparison design, first measurements and what a paid run needs
2. Our harness comparisons so far: zero
23 run records, 3 independent verdicts, not one complete triple
3. First the design, then the empty cells
Five stops: design, measurements, hypotheses, the run and caveats
4. Only the harness changes
The task, agent spec, model and image are pinned in advance
5. An exit code is not a verdict
A record moves through five statuses, and an independent check gives the verdict
6. Two cohorts, with Pi in both
Three harnesses on Sonnet 5.5, three on GPT-6.1 Sol, two tasks, three repeats each
7. Requested is not the same as confirmed
Model identity comes from the provider's response, not the harness config
8. Nine runs, three verdicts, one pass
Pi 0.73.1, local Qwen3.5-9B and task T002 on a Ryzen AI Max+ 395
9. The fails were in committing, not coding
Both agents edited the right files but did not commit before the cap
10. Nine runs, six condition groups
The timeout, the image and the local-model bridge changed between attempts
11. Every harness counts turns its own way
A cap of 30 turns and 2 dollars works differently in five harnesses
12. 44 additions, none confirmed
The harness ledger records every addition and when to retest it
13. Three hypotheses, written before the run
Each has a criterion that would refute it
14. Four conditions block the first paid run
Slot HN-O01: Codex 0.167.0, gpt-6.1-sol, a synthetic Harbor task
15. 24 target dollars to the first comparison
The ladder: one slot, the Harbor pilots, one cohort on T002, the whole campaign
16. The campaign will fill these cells
Per harness: pass^3, the cost of an accepted task and new ledger rows
17. Even a full campaign won't answer everything
Two tasks, three repeats, two models and one date
18. Run first, conclude later
We have zero harness comparisons of our own
Local fails were about finishing the work
Harnesses count turns and dollars differently
The first comparison targets 24 dollars
An empty cell means unmeasured
Next step: a permit for slot HN-O01