Skip to content
Research Insights Made Simple logo
Lab reviewResearch Insights Made Simple

Same agent, different harness: what will runs show?

The harness series finale: the comparison design, first measurements and what a paid run needs

/ Research Insights Made Simple · Agent harnesses 25/25 · Our runs

Slide contents

  1. 1. Same agent, different harness: what will runs show?

    The harness series finale: the comparison design, first measurements and what a paid run needs

  2. 2. Our harness comparisons so far: zero

    23 run records, 3 independent verdicts, not one complete triple

  3. 3. First the design, then the empty cells

    Five stops: design, measurements, hypotheses, the run and caveats

  4. 4. Only the harness changes

    The task, agent spec, model and image are pinned in advance

  5. 5. An exit code is not a verdict

    A record moves through five statuses, and an independent check gives the verdict

  6. 6. Two cohorts, with Pi in both

    Three harnesses on Sonnet 5.5, three on GPT-6.1 Sol, two tasks, three repeats each

  7. 7. Requested is not the same as confirmed

    Model identity comes from the provider's response, not the harness config

  8. 8. Nine runs, three verdicts, one pass

    Pi 0.73.1, local Qwen3.5-9B and task T002 on a Ryzen AI Max+ 395

  9. 9. The fails were in committing, not coding

    Both agents edited the right files but did not commit before the cap

  10. 10. Nine runs, six condition groups

    The timeout, the image and the local-model bridge changed between attempts

  11. 11. Every harness counts turns its own way

    A cap of 30 turns and 2 dollars works differently in five harnesses

  12. 12. 44 additions, none confirmed

    The harness ledger records every addition and when to retest it

  13. 13. Three hypotheses, written before the run

    Each has a criterion that would refute it

  14. 14. Four conditions block the first paid run

    Slot HN-O01: Codex 0.167.0, gpt-6.1-sol, a synthetic Harbor task

  15. 15. 24 target dollars to the first comparison

    The ladder: one slot, the Harbor pilots, one cohort on T002, the whole campaign

  16. 16. The campaign will fill these cells

    Per harness: pass^3, the cost of an accepted task and new ledger rows

  17. 17. Even a full campaign won't answer everything

    Two tasks, three repeats, two models and one date

  18. 18. Run first, conclude later

    We have zero harness comparisons of our own

    Local fails were about finishing the work

    Harnesses count turns and dollars differently

    The first comparison targets 24 dollars

    An empty cell means unmeasured

    Next step: a permit for slot HN-O01