Research Insights #36: Harnesses for Long-running Development (#AI4SDLC)

Materials are available for the solo episode of Research Insights Made Simple from October 9, 2026, “Harness Design for Long-running Application Development.” I examined an Anthropic engineering post about getting an application to a working state when its builder is too willing to declare the job finished. By the way, I covered the paper earlier.

In the episode, I covered:

🔸 Why separate creation from evaluation. A planner expands the request into a specification, a generator builds the application, and an evaluator tests user journeys. A separate evaluator still needs guidance on recognizing material problems.

🔸 When to agree on completion. A sprint contract sets scope and acceptance criteria before implementation. Design also needs clear quality criteria and examples: the final iteration is not necessarily the best.

🔸 What the extra budget buys. In Anthropic’s retro game maker example, a direct run took about 20 minutes and cost $9, but core features did not work. With the harness, it took 6 hours and $200, producing a more complete result. The budgets differ: this demonstrates capability, not universal cost savings.

🔸 What to do after a model change. In the reported move to Opus 4.6, the author tests components one at a time and drops sprints. The value of each role and constraint needs to be measured again.

🔸 What the evaluator misses. In a browser music studio, it can inspect the interface, API, and database, but cannot hear the audio. Checking the Play button does not establish that the music works.

I also discussed the limits of simplification: some harness components compensate for model weaknesses, while others constrain the risk of its actions. A more capable agent does not, by itself, remove the need to control permissions.

Episode materials: 📌 Episode page with timestamps 📊 Slides 📖 Longread on the evolution of agent harnesses 🎬 YouTube, VK Video 🎧 Podster, Yandex Music, Apple Podcasts 📝 Written recap

#AI4SDLC #Agents #Evals #Engineering #ResearchInsights #RnD