How Much Harness Does a Strong Model Need?

A minimal agent loops through a model call, a command and the result. With mini-SWE-agent the model acts mainly through Bash, while OpenCode offers a richer interface. Beyond the Model opens with public SWE-bench Verified results. When the authors compare configurations using the same Claude model, the advantage of richer harnesses tends to shrink on stronger Claude models. Polomodov separates this leaderboard observation from the paper’s controlled comparisons.

The authors run mini-SWE-agent and OpenCode with ten Qwen and DeepSeek models. Across 300 SWE-bench Pro tasks, OpenCode’s lead shrinks from six percentage points with Qwen3-Max to 2.67 points with Qwen3.7-Max. DeepSeek follows a less even pattern. The middle of the range benefits more, while the V4-Pro configurations are separated by two tasks. The paper does not describe repeats or intervals for this experiment, so a small difference needs to be read in light of the sample size.

The authors also count actions. The number of steps Qwen takes in OpenCode barely changes, even as the strongest model makes substantially more tool calls. In mini-SWE-agent, a step is roughly a call, and Qwen3.7-Max takes fewer steps than mid-range models. The authors suggest that a strong model chooses actions more effectively in a minimal interface. The host treats that explanation as a separate question from the observed averages.

A repository bug fix sets a fairly specific goal. GitTaskBench asks agents to use a repository to complete a practical task. ProgramBench asks them to reconstruct a program from documentation and a reference executable. The agent must decide which inputs to probe and when it has enough evidence to implement the behaviour. Strong models can still benefit from a richer harness on this sort of work.

The comparison covers two harnesses and five models per family, with reasoning disabled and temperature set to zero. A team using a different tool interface or allowing the model to reason for longer has a different setup. The host suggests keeping a simple loop that can serve as a baseline when the next model arrives.

To isolate components, the authors add planning, file tools, context compaction, subagents and skills loaded on demand to mini-SWE-agent. Across 200 ProgramBench tasks, file tools and narrowly scoped subagents provide the most consistent individual gains on the two tested models. A subagent may be asked to probe reference behaviour or check discrepancies. The host contrasts these assignments with broad roles such as architect and developer. A narrow assignment makes it clearer what the subagent should investigate and what it should report back.

The complete NanoHarness raises Qwen3.7-Max’s score from 42.88 to 50.25 and DeepSeek-V4-Pro’s from 45.14 to 51.35. On its own, context compaction cuts prompt tokens by about 54% and 77%, respectively, while scores fall by 4.86 and 3.85 points. Enabling all components increases tool calls and prompt length. The host keeps resource use beside the score. Tokens and tool calls show where the amount of work changed, while the final price also depends on how the system is run and billed.

Two common failure modes pull in opposite directions. An agent can keep testing similar inputs against the reference, or it can start coding before it understands the behaviour. In the jq example, NanoHarness cuts reference probes from 133 to 43 while the share of tests passed rises from roughly 25% to 68%. On calcurse, the number of probes and the score both increase. What matters in these cases is which probes the agent chooses and when it begins implementing.

Traces also help on an internal benchmark. If a change improves the mean score, the team can look at the steps where the agent used to stall and see what changed. Polomodov suggests retaining the original action logs for that investigation. The score alone leaves the reason for the difference unresolved.

The companion study uses MLE-bench tasks drawn from Kaggle competitions. The agent must build a pipeline, train a model and evaluate it. Polomodov highlights the move from chat to a session with a shell, files and a compute environment. With the same time budget, this setup lets a strong model compete with specialist systems. Some weaker models benefit more from the support built into a specialist harness.

Polomodov compares the context results across the two papers. In one ML experiment, narrowing Malena’s window to 64,000 tokens with OpenCode’s built-in compaction lowers the estimated medal rate from 57.5% to 50%. The uncertainty intervals overlap. NanoHarness uses a different compaction method and measures a different task. The two experiments motivate checking the cost of losing context in your own workload, particularly for agents that run for hours.

The studies offer different ways to design a comparison. The NanoHarness ablations cover two models on one benchmark. The ML study uses a different set of tasks, models and metrics. The host’s practical advice is to keep a minimal loop, fix the task set and budget, and rerun the comparisons when the model changes. A repeat can show which components still earn their keep, where a component helps only a particular task class, and where it adds work for little gain.