Research Insights #46 Materials: How Much Harness Does a Strong Model Need? (Series #AI4SDLC)

My solo Research Insights Made Simple #46 episode came out on November 9, 2026. In the Harbor episode I looked at how to separate the contribution of the model from that of the harness. “Beyond the Model” asks about specific parts of the agent. Suppose you switch to a stronger model. Is it still worth running subagents, compacting history, and maintaining a separate plan? In “Beyond the Model” and “How Much of a Harness Does a Strong Agent Need for Autonomous ML Engineering?”, the two papers I discuss, the answer depends on what you ask the agent to do.

In the episode

🔸 What changes for bug fixes In the Beyond the Model runs on SWE-bench Pro, OpenCode’s advantage over mini-SWE-agent fell from about 6 percentage points with Qwen3-Max to 2.67 points with Qwen3.7-Max. That result is for those two harnesses. The authors didn’t describe repeats or intervals for these runs.

🔸 Why an open-ended task needs a harness On ProgramBench, the agent has to reconstruct a program from documentation and the behavior of a reference. It decides which inputs to check and when to start writing code. On these tasks, a richer harness also helped the stronger models in the experiment.

🔸 What work to give a subagent On 200 ProgramBench tasks, the NanoHarness authors tested components individually with two models. File tools and subagents with specific work gave the most consistent gains. For instance, one can investigate the reference’s behavior while another checks where the implementation diverges.

🔸 What history compaction costs In the same experiment, compaction on its own cut Qwen3.7-Max’s input tokens by about 54%, while its score dropped by 4.86 percentage points. Judging by token savings alone, the change would have looked like a success.

🔸 What an execution environment adds to ML engineering In the second paper, the biggest gain came from moving from chat to an agent that can read and write files, run code, and see the result. With the same strong model and an equal time budget, one such session was competitive with specialized harnesses in the comparisons studied.

Episode materials

— Episode page with timestamps and slides — Video: YouTube, VK Video — Audio: Podster, Apple Podcasts — Written summary

If you tried simplifying a harness after switching models, tell us what you turned off and how it affected your work.

#ResearchInsights #AI4SDLC #Agents #Evals #Engineering