Research Insights #45 Materials: Model or Harness? (Series #AI4SDLC)
My solo Research Insights Made Simple #45 episode came out on November 9, 2026. When an agent fails a task, what do you tackle first? You can give it a stronger model or look at how the harness gives it tools, runs commands, and collects the result. The authors of “Harbor Adapters and Harbor-Index” compared the contributions of the model and harness across 54 benchmarks. Results and traces are on harbor-index.org.
In the episode
🔸 How to port a benchmark while preserving the task In Harbor, a task uses a common format for instructions, the environment, and result verification. To see whether the adapter changed the test itself, the authors repeated runs in the original and ported versions with the same model and harness.
🔸 Where the 5.2-fold difference between model and harness effects came from They ran each of eight models in its native harness and in the common Terminus-2 harness. After adjusting for benchmark differences, the range of estimated model effects was 5.2 times the range of harness effects. Codex and Claude Code are compared via Terminus-2. A direct measurement would require running the same model in both native harnesses.
🔸 When it’s worth paying for a stronger model In this experiment, medium-difficulty tasks saw the largest score gain. If a more expensive model solves more tasks, compare the cost per successful solution too.
🔸 What a zero in the table can hide The agent might have given a wrong answer or spent tokens without ever submitting a result. Both get the same score. To find where it got stuck, look at its action log.
🔸 How a hard task turns out to be broken In one SWE-bench Pro task, the evaluator required a particular log format that the instructions didn’t mention. Discrepancies like that came up when selecting the separate Harbor-Index set. After the audit, 82 tasks remained. The best result in the third version of the preprint was 28% for GPT-5.5 with Codex.
Episode materials
— Episode page with timestamps and slides — Video: YouTube, VK Video — Audio: Podster, Apple Podcasts — Written summary
If you built your own benchmark for an agent, which task from a work repository would you add first? How would you verify that the agent had solved it?
#ResearchInsights #AI4SDLC #Agents #Evals #Engineering