Harness Design for Coding Agents: What the Harness Contributes

Alexander Polomodov reviews An Empirical Study of Harness Design for Coding Agents as a method for testing individual agent components. The authors keep the ReAct loop fixed while varying planning, the file interface, and context management. Four models are evaluated on SWE-bench Verified and Terminal-Bench 2.1. The main grid covers context; planning and predefined-tool removal are tested only under T4 with a 128,000-token window.

The plan lives outside history and identifies one current step. Context policies preserve history, replace older outputs with placeholders, retrieve evicted events, or summarize. Combining eviction and summarization reduces estimated cost, while context management prevents runs from ending before an attempted repair. Retrieval is rarely used, so external memory alone does not establish a benefit.

Planning extends runs for a weaker model that otherwise gives up quickly, while reducing post-edit verification for stronger models. The savings come with small declines in success rate. Bash-only outcomes depend on both model and workload; Mistral shows opposite directions across the benchmarks. Removing predefined tools also removes their built-in post-edit diagnostics, limiting how the comparison can be interpreted.

Trajectories help distinguish context overflow, early stopping, redundant verification, and invalid tool calls. The host recommends changing components one at a time, repeating runs, and retesting the harness after a model change. Single runs per task, a limited model sample, and SWE-bench Verified issues prevent the settings from being universal recommendations. Timestamps follow the Podster audio version and may differ slightly from YouTube.