Harness design for coding agents: what does the harness contribute? (#AI4SDLC)

I read this September’s “An Empirical Study of Harness Design for Coding Agents” with great interest. Its authors tried to determine which parts of a harness help and by how much: planning, tools, or context management. They focused on those three factors.

The authors use the conventional definition of a harness: an environment that gives a model tools, supports its workflow, and determines which history it sees. Researchers from UMass Amherst, Emory, UNC Charlotte, and Zoom built a modular harness and varied components while keeping the execution loop fixed.

The paper is currently an arXiv preprint, with a Hacker News discussion, and has not yet been submitted to a major conference. Its methodology is nevertheless substantial:

1. Four models: Nemotron-3 at 30B, 120B, and 550B, plus Mistral Medium 3.5; 176 configurations and 32k–128k windows.
2. Two benchmarks measure task success and cost: SWE-Bench Verified, with 500 tasks from Python repositories, and Terminal-Bench 2.1, with 89 terminal tasks.
3. Paired comparisons on the same tasks: McNemar’s test considers discordant successes and failures. Benjamini–Hochberg limits the expected proportion of false discoveries across multiple comparisons.

The authors’ main results are roughly as follows.

1️⃣ Context management primarily lets an agent continue working On SWE-Bench, the average gain of managed strategies over T0, with no context management, falls from 35.7 percentage points at 32k to 2.7 at 128k. T0’s overflow rate drops from 78.7% to 8.7%. Trajectories suggest the agent behaves similarly but has time to reach editing and verification. This does not establish an improvement in reasoning itself.

2️⃣ Cheap history cleanup before summarization offers a good balance T4 first removes large old outputs, then summarizes the history if necessary, so expensive summarization calls are needed less frequently. Averaged over four window sizes, it has similar success and the lowest mean cost among managed strategies in seven of eight model–benchmark pairs. It does not win on success in every configuration.

3️⃣ Saved memory is barely used Adding retrieval of old messages, T2 versus T1, brings no consistent improvement: the mean difference is −0.36 percentage points. In 36 of 64 T2/T4 configurations, recall_event was never called. Its own description suggests rereading the file or rerunning the command. This is a result about a particular retrieval mechanism, not proof that external memory in general is useless.

4️⃣ Planning helps a weak model continue and a strong model finish in time For Nemotron-3 30B, SWE-Bench success rises from 13.6% to 25.2%, along with cost. Without planning, 68.6% of runs end without an edit; with planning, 27.8% do. For 550B and Mistral, trajectory analysis shows less repeated verification: estimated cost falls by about 30% and 32%, while success falls by 2 and 0.4 points. The same component helps address different problems.

5️⃣ Shell can make actions larger and cheaper, but the result depends on the task For 550B, switching to bash-only on SWE-Bench raises success from 65.8% to 69.4% and lowers cost from $2.33 to $1.11. Edits are larger and interactions fewer. Mistral’s success on the same benchmark falls from 68.6% to 45.4%, while on Terminal-Bench it rises from 37.08% to 43.82%, together with cost. Model size alone does not determine the right interface.

I marked the limitations in red: one run per task; planning and tools were tested only at 128k with one context strategy; instructions and automatic checks changed alongside the tools. Two model families are still a narrow basis for universal prescriptions.

The authors also did a good job analyzing agent trajectories to interpret the results. That is worth adopting in our own experiments.

On Tuesday, October 13, we will discuss this whitepaper live on Research Insights Made Simple.

#AI4SDLC #Agents #Evals #Engineering #SystemDesign #Software