Harbor: How to Tell Why an Agent Failed a Task (Series #AI4SDLC)

I’d call Harbor a USB hub for benchmarks. Suppose you have m agents and n benchmarks. Connecting each agent to every benchmark can take up to n × m integrations. With a common interface, each agent and each benchmark connects to Harbor once. That gives you n + m :)

In the Harbor Adapters and Harbor-Index preprint, Lin Shi, Haowei Lin, and their coauthors did three things.

1️⃣ They moved more than 80 benchmarks to Harbor’s common format. Each task has instructions, an environment, a check, and a reference solution. The authors compared the adapters’ results with those of the original implementations.

2️⃣ They ran 8 models on 6 627 tasks from 54 benchmarks. You still have to run every combination you want to evaluate. For each model they compared the common Terminus-2 harness with the harness from its developer, running each configuration three times. By their count, the experiment used 226 billion tokens and cost over $300,000 in compute. Benchmark runs aren’t cheap these days.

3️⃣ They built Harbor-Index 1.0 from 82 tasks across 29 benchmarks. They started with tasks that strong models rarely solved. Then they checked them automatically and manually, fixed some, and reran them. The manual audit ruled out about a third of the hard candidates because of defects. In a separate experiment on the index, the best result in the paper was 28% for GPT-5.5 with Codex.

GAIA2 gave a clear example. On four tasks that reached the manual audit, the adapter never sent the events that the evaluator was waiting for. Whatever the agent did, the run ended in a timeout.

What I noted in the margins

🔸 How many benchmarks do you need to tell models apart? The authors picked 12 benchmarks. For each of the others, at least one of those 12 gave a similar system ranking. Within a single benchmark, sometimes three carefully chosen tasks are enough. But they chose the tasks based on the results of the same systems. I’d be interested to see whether the ranking holds for the next generation of models.

I also noted the result about harnesses. In the large experiment, the variation associated with model choice exceeded the variation associated with harness choice. Yet in the index table the same GPT-5.5 got 19,5% with Terminus-2 and 28% with Codex. One caveat about the Native Harness column: for six models accessed through OpenRouter, it lists Claude Code.

🔸 I’d calculate the cost of a successful solution In the chart, stronger models use fewer tokens, but an attempt costs 2–3 times as much. I used Fig. 3 to estimate what happens when you divide the cost of an attempt by its success rate. In the hardest bucket, strong models succeed 4% of the time, compared with 1% for the others. With that difference, pricier attempts can yield a cheaper solution. This is a rough calculation. The rates are rounded, and task difficulty was estimated from the same runs.

🔸 I kept the checklist in Appendix D.7. I used it to check one of my coding-agent evals. In one control case, a working solution received fail because Node couldn’t start tests at the given path. In another, the evaluator accepted three empty tests because it counted passes without checking test contents. After the fix I repeated the controls. The working solution passed, the code error showed up in the failed test log, and the substituted tests were rejected. My most useful takeaway from the paper was to run a reference solution and a known bad submission through the evaluator and see what it credits.

P.S. On Monday we’ll talk about the whitepaper in more detail on a live Research Insights Made Simple episode. I’m thinking of trying the index with one of the Chinese models over the weekend and reporting on the stream how easy it is.

#AI4SDLC #AI #Agents #Evals #Research #Engineering