Model or Harness: What Moves the Score?

Benchmarks have different ways of handing an agent a task and checking its work. With a bespoke connection for every agent and benchmark, the number of integrations grows as the product of the two counts. Polomodov compares Harbor to a USB hub. Port each benchmark once, connect each agent to the shared infrastructure once, and new combinations become possible without writing another point-to-point integration.

A Harbor task defines instructions, an execution environment, and a check on the final state. A conventional question can become an agentic task by giving the agent files and tools to gather information and act. That move needs a parity check. Changes to a prompt template, a base image, or a tool can move the score even with the same model.

The researchers run original and ported benchmarks under matched settings and repeat the trials. The episode covers both close results and a small FrontierCS example with a wider gap. Repetitions help put a porting difference in the context of ordinary run-to-run variation.

The main experiment runs eight models across 54 benchmarks. Each model uses its native harness and the shared Terminus-2 harness. After accounting for different benchmark baselines, the estimated model effects span about 45 percentage points, while the harness effects span about nine. The ratio of those ranges is 5.2. It describes variation within the configurations the researchers studied.

The host points out what the shared harness lets the researchers compare. GPT in Codex is compared with GPT in Terminus-2. Claude in Claude Code is compared with Claude in Terminus-2. Terminus-2 connects the groups. A direct Codex versus Claude Code comparison for a single model would require running that same model in both native harnesses. The estimated range is a starting point for testing the specific combination a team intends to use.

The authors group tasks by how frequently the tested configurations solve them. Cheaper models handle many easy tasks. At the other end, even the strongest models solve very few. The episode illustrates this with an increase from roughly 1% to 4% on the hardest tasks. That is a large relative gain and only a few additional solves. Tasks in the middle of the range leave more room for a strong model to make a practical difference.

A higher price per attempt can still produce a comparable cost per solved task if the agent succeeds more often. Trajectories help explain where the spend goes. An agent may implement a wrong algorithm, return an incorrect answer, or do some useful work and fail to submit it. The last case can look like any other failure in the aggregate score, even though it points to an execution problem worth investigating.

Harbor-Index is a separate experiment with 82 tasks. In the initial screen, three models try each candidate in two harnesses, with three trials per configuration. Tasks advance if they are solved at most six times out of 18. An auditor and human reviewers then examine instructions, verifiers, and trajectories. The final set draws from 29 benchmarks.

A low solve rate can come from a faulty task. In one example, the instructions ask for a bug fix and the verifier also expects a particular log format. An agent can fix the bug and still score zero. Reviewers repair disagreements of this kind. Some tasks become too easy for the index after repair. The best score on the final set in the third preprint version is 28%. The models and task set differ from those used to estimate the effect ranges.

Harness choice also affects cost on the index. The preprint prices the GLM 5.2 run through Claude Code and OpenRouter at $205 for seven solved tasks, versus $52 for eight in Terminus-2. A one-task difference is weak evidence for a difference in quality on a suite of this size. The cost difference matters for budgeting repeat runs. The authors flag low cache-hit rates as a possible explanation, so teams should check costs under their own model access setup.

Two configurations can have similar scores and solve different tasks. The episode discusses Terminus-2, which accesses its environment through Bash. A task that depends on an image or a search can play out differently when a dedicated tool is missing. The timeout also belongs to the evaluation setup. A tighter limit can stop a run that would finish with a larger budget.

A listener asks how to compare agents on 1C tasks. Polomodov suggests creating a verifiable suite for that domain first. Harbor-Index offers a way to organise the experiment, while the team’s own tasks determine what a good result looks like. The episode also compares the early Harbor-Index blog post with the third preprint version. Their numbers differ, which makes the source version, model, harness, and run conditions worth keeping alongside a reported score.