Measuring Coding Agents in Dialogue

A conventional coding benchmark gives an agent a complete specification and checks the final repository state after a long autonomous run. This is convenient for comparing models but unlike much real development: the initial request may be incomplete, constraints surface after the first changes, and the user clarifies intent after noticing a wrong direction. A multi-turn evaluation therefore measures execution together with the ability to preserve context and incorporate corrective feedback.

SWE-Together works bottom-up. Its authors convert real user sessions into 109 reproducible tasks and build a reactive simulator that compares the original intent anchors with the agent's current trajectory. SWE-INTERACT works top-down: it selects 75 existing benchmark tasks, keeps their original verifiers, and reveals hidden requirements in small increments through an exacting senior-engineer persona. The first design is closer to observed practice; the second isolates the single-turn versus dialogue gap more directly.

SWE-Together separates outcome from user effort. A frozen rubric answers whether the task was completed, while User Correction counts redirects and softer nudges. Across seven evaluated models, User Correction has a strong negative relationship with pass@1: Pearson −0.92. Stronger models generally require fewer interventions, but the episode argues that token price is incomplete economics: every correction consumes an engineer's attention to observe the work, diagnose deviation, and formulate the next turn.

SWE-INTERACT shows an even sharper effect. On identical tasks with the same verifier, GPT-5.5 resolve rate falls from 48.0% to 24.7%, while Opus 4.8 drops from 50.7% to 26.7%. GPT-5.5 uses up to 3.9 times as many steps and costs 3.5 times more. Technical implementation bugs and forgotten requirements dominate failed trajectories: an agent may discover a constraint early in the conversation yet omit it from the final implementation.

The speakers do not treat these results as a universal model ranking. SWE-Together retains only 0.97% of the source sessions and uses LLMs for both simulation and judging. SWE-INTERACT relies on one persona, provider-specific harnesses, and a user simulator with shell access that can inspect work faster than a person. Each result therefore describes a particular model × harness × simulator × verifier system, and production transfer requires local repositories and scenarios.

A practical scorecard should retain final success, User Correction, and the single-turn-to-multi-turn gap for the same tasks. An explicit requirement ledger, an upfront plan, per-iteration commits, and independent verification may reduce forgetting and make review cheaper, but these are hypotheses for an A/B evaluation rather than proven remedies. The essential shift is to evaluate an agent as a collaborator: can it ask, preserve agreements, absorb corrections, and leave evidence that a human can verify?