Agent harnesses: what actually helps
Planning, tools and context: a controlled comparison
Slide contents
1. Agent harnesses: what actually helps
Planning, tools and context: a controlled comparison
2. An agent ranking hides several causes
3. Harnesses can be tested part by part
Planning · actions · context around one loop
4. 176 settings leave some combinations untested
4 models × 2 benchmarks × 22 settings
5. The two benchmarks exercise different work
Repository repair differs from terminal task completion
6. The plan lives outside conversation history
update_plan → stored plan → next model input
7. The whole action interface changes
Predefined operations ↔ bash commands
8. Five policies combine three mechanisms
Elision · recall · summarization
9. Cheap elision comes before summarization
T4 preserves the task preamble and recent tail
10. Smaller windows make context management more valuable
Success rate by window size, both benchmarks
11. Most gains come from avoiding overflow
T0 overflow failures and trajectory length
12. T4 is the cheapest managed tier
T4 triggers costly summarization less often
13. Models rarely recall output behind placeholders
T2 vs T1: recall added, all else equal
14. Planning helps the weaker model persist
Nemotron-3 30B · SWE-bench · T4/128k
15. Planning trims redundant checks for stronger models
Nemotron-3 550B and Mistral Medium 3.5 · SWE-bench · T4/128k
16. Bash helps one model and hurts another
T4/128k · planning enabled
17. One model changes preference across benchmarks
Mistral Medium 3.5 · T4/128k
18. Each component leaves its own mark
Trajectory length · stopping point · action size
19. Behavior annotations need validation too
LLM judge + a human validation sample
20. Results depend on models, tasks and implementation
Four models · two families · one attempt per task
21. Symptoms point to components
Our synthesis from the study
22. A harness must earn its cost
Context management removes window limits
Elision before summarization cuts cost
Planning changes when agents stop
Interfaces depend on model and workload
Recheck value after model changes
Measure the failure, then change the mechanism
23. The exam itself needs scrutiny
24. Next, test interactions between components