Measuring Coding Agents in Dialogue

Aleksey introduces himself: over 13 years in IT, time spent as both an engineer and a manager, a long stretch running a development department of more than a hundred engineers where he built the engineering strategy and hiring, and now work on rolling artificial intelligence out to large and mid-sized companies, a training programme and a book he is preparing for publication in Russian and English. Alexander liked several things about this pair of papers at once: they appeared on the same day, from two connected labs — Meta and a company whose stake Meta bought, whose former head now runs Meta's AI direction — and they approach a similar topic from opposite sides. The idea, both agree, looks obvious in hindsight: benchmarks used to measure the case where you dump the whole specification, the agent goes off and works autonomously, and the result is checked at the end, but people do not work with agents that way. Aleksey adds a reader's impression: SWE-Together is packed so tightly that it reminded him of an old hardcore algorithms textbook, while the second paper reads far more easily.

The participants then separate the terms. Single turn is the giant prompt after which you simply wait to see whether the model finished the job, and finished it in the shape expected in advance. Aleksey notes that the dream of throwing in one prompt and getting the result straight away lives mostly with business people far from engineering, while engineers fall into the opposite bias and approach everything far too iteratively. Multi-turn is when you answer the agent bit by bit, unblock it and feed requirements as you go. The first paper, Aleksey recalls, describes cases where top labs stop taking part in benchmarks altogether: the agent genuinely solves the task, but the check looks too narrowly and refuses a correct solution that was reached another way. Alexander also asks the audience for a good Russian term for single turn, since the usual renderings sound clumsy, and promises the author of a good one a separate stream.

SWE-Together is built bottom-up: real sessions were turned into a little over a hundred reproducible tasks. Alexander walks through the funnel on a slide: a session needs enough interactions and actions to show that an agent, not a human, did the work; the task has to be reproducible and gradeable; and an environment with a verifier has to be assembled around it. That is hard engineering work, and it also explains why more than 99% of real sessions are thrown away. The sample comes out very clean but not random: if a task needs CI/CD knowledge or an integration with an external system, it simply is not measured, and it is unclear what population the result generalises to. Aleksey asks why Meta, with all its own data, collected open source instead; Alexander plays the other side — on internal data the benchmark could not be interpreted outside Meta's own context. Both would like to see the same methodology applied to internal tasks and at least published.

SWE-INTERACT goes top-down: 75 tasks, 25 from each of three existing benchmarks whose names are hard to make out in the captions, with two of the three frameworks belonging to the authors themselves and the original verifiers kept in place. That gives what Meta lacks: a one-shot baseline against which you can see how much releasing information in pieces degrades an agent. Requirements are revealed by a nitpicking senior micromanager, one of the user personas the authors derived by clustering sessions, who feeds precise, directive but incomplete instructions. The simulators differ: Meta's sees the agent's trajectory of actions, which is closer to a human watching reports in a console, while SWE-INTERACT's simulator has a shell, greps files and inspects diffs — no human keeps up with that, and the participants would cap its cognitive bandwidth, otherwise this is an agent-to-agent relationship. Alexander also recalls the half-century-old principal-agent problem in economics: the principal pays for the outcome, the agent wants rewarding for activity and tends to dig only the two ends of the trench, since those are what get checked. Aleksey adds that this problem, he thinks, won the 2016 Nobel prize, with a proof that it cannot be solved optimally.

SWE-Together splits evaluation into three axes: final correctness, preservation of the original intent, and user correction, the count of times the user steers the agent. Alexander says he had hardly seen such a decomposition before: a benchmark usually answers solved or not solved, sometimes with a degree of correctness assigned by an LLM judge. The headline result is that stronger models need fewer corrections, with a relationship coefficient of almost −0.92. Aleksey converts that into money: user correction is the cost of a person's time, because you cannot know in advance which task was done wrong and cannot skip checking; the fewer corrections, the sooner you switch to something else. What the paper lacks, he regrets, is a comparison of the engineer's attention cost with the model's price. Alexander recalls a Cursor report from April or May, as he remembers it, where one model was roughly eight times cheaper than the GPT models and some sixteen times cheaper than Opus while losing only 3 to 5 percentage points — but only single-turn work was measured.

In SWE-INTERACT both Opus 4.8 and GPT-5.5 were nearly twice as strong in the one-shot setting: with requirements revealed gradually, success roughly halves, the number of steps grows up to fourfold and cost rises about 3.5 times. Sonnet, by Aleksey's recollection, barely suffered — around 27% against 24 to 25% — which gives him a practical conclusion: if you are going to micromanage the agent anyway, save money on the model. The frontier models fell hardest, having been tuned to work long on a single task because benchmarks asked for exactly that. In the newest models he sees overengineering: with no explicitly stated minimum expected change, the agent does far more than asked. Among failed trajectories 34% are technical implementation errors, followed by forgotten requirements, though GPT-5.5 did not forget them in more than 90% of cases. Aleksey closes on the economics of verification: AI made producing code cheaper but not checking it, and the hour of attention it takes to review 500 lines has not gone anywhere, because plausible and correct are indistinguishable until you check.