Measuring Coding Agents in Dialogue — with Aleksey Litvinov

Alexander Polomodov and Aleksey Litvinov compare SWE-Together and SWE-INTERACT, which test coding agents as requirements emerge in dialogue. A conventional benchmark sees the final repository but not whether an agent preserves context or how much user attention a wrong path consumes.

SWE-Together turns real sessions into 109 tasks and uses a reactive simulator. SWE-INTERACT selects 75 tasks, keeps their verifiers, and reveals hidden requirements through a senior-engineer persona. The first is closer to practice; the second isolates the gap between a full specification and dialogue.

In SWE-Together, User Correction correlates with pass@1 at Pearson −0.92. In SWE-INTERACT, GPT-5.5 falls from 48.0 to 24.7 percent and Opus 4.8 from 50.7 to 26.7 percent; GPT-5.5 also takes up to 3.9 times more steps at 3.5 times the cost. Forgotten early requirements are a common failure.

These are not universal rankings: SWE-Together retains 0.97 percent of sessions and uses LLMs as simulator and judge, while SWE-INTERACT depends on one persona and provider-specific harnesses. Internal scorecards should track final success, corrections, and the multi-turn gap; requirement ledgers, planning, and independent verification remain A/B-test hypotheses.