What a Coding Agent Gains from Its Harness

The review opens with a question about causality. When two complete agents achieve different scores, the comparison does not explain whether the model, plan, tools, or history handling made the difference. The authors build a lightweight harness around a ReAct loop: thought, action, observation. They vary three surrounding components while keeping turn limits, generation settings, and other conditions shared. This makes it easier to connect a change to observed behavior.

The study compares 176 settings across four models and two benchmarks. SWE-bench Verified supplies 500 Python repository repair tasks; Terminal-Bench 2.1 supplies 89 command-line tasks. The models are three sizes of Nemotron-3 and Mistral Medium 3.5. The main grid varies context policy and window size. Planning and file-tool removal are tested only with T4 context management and a 128,000-token window, so the study does not exhaust all component combinations.

The plan lives outside conversation history. On the first turn, the harness reminds the model to create one; subsequent requests include its current state. The agent can update steps, with exactly one marked in progress. This provides a structure for moving through the task. Without the plan, the model decides how to break down the work using its instructions and available history.

A separate setting controls available actions. The full interface includes bash plus dedicated file-reading, search, and editing tools. In the alternative, file operations must go through bash. One caveat matters: post-edit diagnostics are built into the editing tool and disappear with it. The comparison therefore changes feedback as well as command convenience. Planning and history-retrieval mechanisms are considered separately from the file interface.

The authors compare five context policies. T0 keeps history without compression. T1 replaces older outputs with placeholders; T2 also lets the agent retrieve evicted events from external storage. T3 summarizes, while T4 combines eviction, retrieval, and summarization. System instructions, the original task, and recent turns remain available. The middle of the history can shrink while preserving the objective and the immediate context for continuing work.

Under the combined policy, eviction begins at roughly 60% of the window and summarization at 85%. Summarization is therefore not the first response to a growing history. Active management helps particularly with small windows: otherwise, a run can end before the agent even attempts a code change. Trajectory analysis attributes much of the improvement to preventing that interruption, rather than demonstrating that compression itself improves reasoning.

The combined policy has the lowest average estimated cost in the comparison discussed. Evicting old outputs before summarizing the remaining history can cost less than repeatedly summarizing a larger body of text. Yet the cheapest policy does not achieve the highest success rate in every setting. Both measures must be considered before selecting a configuration.

Agents rarely retrieve evicted information. Smaller models with tight windows use the capability more often. External storage alone does not ensure that a model will notice missing information and request the relevant event. Measuring actual retrieval calls and their outcomes is more useful than assuming that recoverability must help. A mechanism can look compelling in an architecture diagram while contributing little to the observed work.

For the weaker Nemotron-3 30B on SWE-bench, planning increases median run length from five to forty turns. The model abandons fewer runs before editing, gets stuck less often, and solves almost twice as many tasks. Planning helps sustain work: explicit steps provide a direction the agent can follow. Additional turns are useful because the previous behavior ended too early.

Stronger models show the opposite pattern. Nemotron-3 550B drops from 108 to 74 median turns, and Mistral from 68 to 53. Less work goes into post-edit verification, reducing estimated cost. Success rates nevertheless decline slightly, by 2 and 0.4 percentage points respectively. Savings are not automatically a quality improvement. Plan format and stopping behavior remain candidates for further experiments, while small differences call for repeated runs.

Predefined tools help the smaller Nemotron, whose success rate falls sharply with bash alone. The larger Nemotron moves in the other direction, combining operations into scripts and accomplishing more with each call. The interface changes the size of an action, so turn counts cannot be interpreted independently of what each turn does. A shorter trajectory can mean either premature abandonment or effective command composition.

Mistral contradicts the rule that strong models only need bash. Dedicated tools help on repository repairs: 32.8% of bash-only runs finish without an edit, compared with roughly 1.2% using tools. Terminal tasks show the opposite direction; even with the full interface, almost 72% of actions already go through bash. Results depend on the model and workload together. A model change should prompt another comparison of tools, context settings, and feedback under otherwise comparable conditions.

The authors use GPT-5.5 to analyze action sequences and compare its labels with human checks. The purpose is to move from a score to a cause: context overflow, stopping before editing, repeated verification, or calls to nonexistent tools. Each symptom can motivate a specific hypothesis about a component. For example, a planning experiment can measure whether fewer runs end before the first edit.

The host connects this method to the earlier Anthropic review: changing several components at once makes regressions difficult to explain. A practical sequence is to establish a baseline, change one component, and examine both the outcome and the actions leading to it. Telemetry helps choose the next experiment. It does not replace checking the final solution; fewer actions can still produce an incorrect patch.

The study uses one run per task at temperature zero. Models run locally through SGLang, while costs are estimated using August 2026 OpenRouter prices. These are token-based estimates, not measured local hardware bills. Four models from two families, a small Terminal-Bench sample, and narrowly scoped component-removal tests limit transferability. Temperature zero does not measure variation across repeated runs.

The closing discussion considers SWE-bench Verified limitations: some checks can reject functionally correct solutions, and tasks or answers may appear in training data. For an internal system, the host recommends a relevant task set, repeated runs, and broader testing of component combinations. The paper offers a repeatable way to question harness design. After a model upgrade, each mechanism should demonstrate its value again through result quality, cost, and observed failure causes.