Harness design for long-running autonomous application development (#AI4SDLC)

In episode 33 of Research Insights Made Simple, we explored SWE-agent: how the interface, tools, and feedback change the results of the same model. In episode 34, we looked at handing work between sessions while retaining a feature list, progress, and a verified project state. In episode 35, we discussed OpenAI’s approach to harnesses. In “Harness design for long-running application development,” Prithvi Rajasekaran describes Anthropic’s approach to the same problem. In effect, this moves from the agent’s workspace and memory to organizing autonomous, one-shot development of an entire application.

LLMs can easily fall into mutual self-congratulation when evaluating themselves: write something, check it, and decide it is all excellent. The author separates creation from evaluation. The full harness has three roles: 1) Planner expands a short request into a product specification. 2) Generator implements the application. 3) Evaluator checks the running system through the browser, APIs, and database state.

To me, this already looks a lot like a team with a product manager, a developer, and QA. The sprint contract is particularly interesting: before writing code, the agents agree on what should be delivered and how to verify completion. The specification remains fairly broad, while acceptance criteria connect it to implementation.

A separate critic also needs tuning. The author examined its logs and taught it to take detected problems more seriously. For design, he defined criteria for coherence, originality, execution quality, and usability, then calibrated scores using examples. I am interested in the boundary here: how far can taste be formalized? An evaluator may become increasingly accurate at matching specified preferences, but higher scores do not guarantee that a person will prefer the next version to the previous one.

The cost is noticeable too. In the game editor example, a single-agent run cost $9 and took 20 minutes; the full harness cost $200 and took six hours. According to the author, the second version had a working game mode. But the budget, time, and task scope grew together, so this comparison does not isolate the contribution of separating roles.

Then comes the most useful part for engineers: simplifying the harness as the model improves. With Opus 4.5, the author could remove explicit context resets; with 4.6, he removed sprints, retaining checks after the build and correction cycles. It is the familiar turn from SWE-agent’s history: parts of a harness can stop being necessary.

Each element captures an assumption about a model’s weakness. After an update, it is worth checking again why we need that handoff, that role, or that intermediate stage. Remove them one at a time and observe quality, time, and cost.

My next question is whether an agent could improve its own harness: change the process, verify the result, and remove unnecessary pieces. It sounds like a natural continuation, but then it becomes especially important to establish who evaluates the evaluator.

On October 9 at 09:00 Moscow time, we will discuss this article live on Research Insights Made Simple. We will explore the process, the value of independent acceptance, and what remains of the harness as models improve. Join us!

#AI4SDLC #Agents #Engineering #Research #Evals