Beyond the Model: When a Harness Helps an Agent

The abstract of “Beyond the Model: Demystifying Harness Effects in Software Engineering Agents” opens with an interesting qualification. As models get stronger, elaborate harnesses help less with bug fixing. On more open-ended tasks, stronger models may actually make better use of what the harness offers. Here a harness comprises tools, context management, planning, and other mechanisms around the model that shape how the agent works with a repository and checks its solution.

A team from Nanjing University of Science and Technology, Nanjing University, and TU Munich compared mini-SWE-agent with OpenCode across ten Qwen and DeepSeek models. They used three benchmarks: “SWE-bench Pro” for bug fixes, “GitTaskBench” for tasks that use repositories, and “ProgramBench,” where the agent must reconstruct a program from documentation and the behavior of a reference executable. The preprint is dated September 26, 2026.

The authors then built NanoHarness on top of mini-SWE-agent and tested its components separately and together on all 200 ProgramBench tasks, this time with two models.

1️⃣ Tools and specialized subagents deliver the most consistent gains in these experiments
The tool layer has just four operations: read_file, write_file, edit_file, and glob. Subagents dedicated to analyzing behavior, comparing implementations, and checking results help more than broad roles like “architect” or “developer.” Planning and on-demand skills show smaller or model-dependent effects.

2️⃣ Context compression saves tokens at a cost to quality
The two models use about 54% and 77% fewer input tokens, while their scores drop by 4.86 and 3.85 points. The authors suggest that relevant details may be lost. This is one implementation on ProgramBench, so it's too soon to generalize the result to context management as a whole.

3️⃣ The full harness outperforms every individual addition
Qwen3.7-Max goes from 42.88 to 50.25 and DeepSeek-V4-Pro from 45.14 to 51.35. NanoHarness comes close to OpenCode and Claude Code. The comparison doesn't establish whether every component is necessary.

The agent's behavior is interesting too. It can spend ages investigating minor details or decide too early that it understands everything. In the jq example, NanoHarness brings checks against the reference program down from 133 to 43, while the share of passing tests goes from 24.68% to 68.17%. In another example, both the number of checks and the result improve. The authors attribute this to more deliberate exploration of the task. The examples illustrate their explanation.

A few of my notes in the margins

🔸 The claim that harnesses hadn't previously been studied systematically seems debatable. I'd like to pin down what's novel and separately work out how far ProgramBench lets us generalize.
🔸 Why test every model in non-thinking mode? The authors explain setting temperature to zero as a way to reduce randomness, but this choice is less clear. I'd also like to see cost per task next to the score.
🔸 The question I find most interesting is whether models are fine-tuned for the minimalist mini-SWE-agent and for working through Bash. If so, a narrowing gap between harnesses may partly reflect familiarity with that interface. The authors allow for training on SWE tasks, but training for a task and training for a way of acting are different. How much is general model ability and how much is familiarity with a particular harness?

P.S.
We'll discuss this whitepaper on Thursday in the Research Insights Made Simple live stream, and I've moved today's stream on “On Randomness in Agentic Evals” to noon.

#AI4SDLC #Research #Agents #Evals #Engineering