Slide contents

1. 1. How much harness does a strong model need

   When a harness helps and when a minimal loop suffices

2. 2. It depends on task and model

   Four stops: leaderboard, whole harnesses, components, ML engineering

3. 3. #38's question, on ten models

   Fan et al. varied three mechanisms inside one light loop

4. 4. Harnesses rival models in effect

   SWE-bench Verified leaderboard: harness +19.4 pp, model +22.8

5. 5. Stronger model, smaller harness gain

   Gain over mini-SWE-agent on the same model, SWE-bench Verified

6. 6. Each cell is a single number

   Two harnesses × 10 models × 3 benchmarks = 60 configurations

7. 7. The gap narrows but does not vanish

   Qwen on SWE-bench Pro (N = 300): OpenCode's lead 6.00 → 2.67 pp

8. 8. Stronger model, more tool calls

   OpenCode on Qwen: 36–38 steps, tool calls 39.87 → 72.23

9. 9. DeepSeek's gain has a threshold

   SWE-bench Pro: little on R1, more on V3–V3.2, almost none on V4-Pro

10. 10. Mostly the strongest models gain

    Open tasks: only the strongest models gain consistently

11. 11. Components switch on one by one

    NanoHarness: five minimal modules on top of mini-SWE-agent

12. 12. Some components hurt

    ProgramBench, 200 tasks: score change relative to mini-SWE-agent, pp

13. 13. Five components nearly catch up

    84–85% of OpenCode's gain (our calculation); 1.1–2.1 pp behind

14. 14. Compression cuts cost and score

    Tool calls, prompt tokens and score vs mini-SWE-agent

15. 15. The harness evens out probing

    Two mini-SWE-agent failures: too many probes and too few

16. 16. In ML, environment matters most

    Companion, MLE-bench: from chat to several agents on an equal budget

17. 17. Malena matches four systems

    GLM 5.2, 30 MLE-bench tasks, 24 hours on one A100

18. 18. Weak models still need a harness

    Percentile difference between Malena and the strongest rival, paired 95% interval

19. 19. The two papers agree in two places

    Context compression and delegation agree; skills diverge

20. 20. Components were tested narrowly

    Harnesses on three benchmarks, components on ProgramBench only

21. 21. Only “no worse” is solid

    Four limitations the authors name and what was done

22. 22. Minimum first, then ablations

    A working order from both papers: the host's view