SWE-Interact: how much does interactivity cost coding agents (Category AI4SDLC)

I decided today to talk about the second article about the evaluation of coding agents in the dialogue.SWE-Interact: Reimagining SWE Benchmarks as User-Driven Long-Horizon Coding Sessionsfrom Scale AI. A curious coincidence: she came out on arXiv 29 June 2026 Same day as Meta's SWE-Together (banned)which I am today handler. The two teams almost simultaneously came to the same idea: single-mode benchmarks ceased to reflect the real work with agents. It seems that the multi-turn assessment is issued in a separate direction. (Meta and Scale AI are related. ||Oh, I'm sure Meta bought it. 49Scale AI Percentage for $14 Billions and more took the founder of the company||)

Scale collects benchmarks from the opposite side. Meta goes from bottom up: from 11,260 Real sessions are restored 109 tasks and anchors the simulator on the intentions of live users. Scale goes top-down: takes 75 Ready tasks from three existing benchmarks (SWE-bench Pro, SWE Atlas (Refactoring)DeepSWE - the first two, I note, her own)It hides the full staging inside the user’s simulator and forces the requirements to be disclosed gradually. The solution checks the original problem verifier. The move is elegant: the same task, the same verifier - only the way claims are submitted changes. The difference in resolve rate becomes almost the net cost of interactivity.

And then the price was really high - the best models decide about it. 50Percent of tasks in single-mode mode, and only about a quarter of the same tasks in dialog:5.5 fall off 48.0percentage 24.7%, Opus 4.8 - 50.7percentage 26.7%. The steps are usually in 3–4 times more, tokens - up to 4.5 and the cost of the task in 2–3.5 higher (opus 4.8 - $11.80 against5.09). Working with the user adds a separate axis of complexity to the task.

The user simulator here is more interesting than the Meta 1Authors created a persona from statistics To do this, they analyzed real sessions from the SWE-chat dataset and modeled the most frequent type - a meticulous expert in vibecoding mode: a senior developer who writes short and carelessly, does not touch the code himself. (According to SWE-chat, in this mode, the agent writes more 99percent)but meticulously follows the exact signatures of the API and issues comments one at a time 2The simulator itself is an agent He has shell access to a working copy of the agent. (git, grep, sed, find), and before answering, he watches diffs and commits himself. In Meta, the simulator sees only text - trajectory summaries and tool conclusions; the authors of SWE-Together themselves listed this among the limitations. And here the experiment shows that a meticulous persona is pain: with it, the steps of the trajectories in most models grow by a certain amount. 30–50%, the number of user replicas almost doubles, and the resolve rate decreases 4 from 5 models compared to a neutral simulator that simply issues requirements on request.

The article also contains an analysis of failures. (It's an audit. 287 trajectories). Some models know how to ask well and already on a vague description lay in the plan more 80Percentage of hidden targets and GPT-5.5, Opus 4.8 Sonnet 4.6 At the end of the dialogue, more 90%

• Failures are grouped as follows: Technical Errors of Implementation (~34percent)
• Forgotten claims (~34%) When a demand from an early replica never made it to the final code
• In ~12The simulator itself did not issue the necessary requirement, but this is most likely a false positive of the benchmark itself, as the authors say in the article.

There are research and limitations

• The task is complete. 75 (Meta 109 task)
• There's only one person. (It would be interesting to model different users.) Single-track baseline run once The result significantly depends on the model of the simulator: with GPT-5.5 as an Opus user 4.8 decider 30.7%, with Opus 4.7 - 22.7%
• Agents worked in their own home bandage. (harness) - Claude Code, Codex CLI, Kimi CLI, and Gemini in third-party OpenCode - so the comparison of models is mixed with the comparison of bindings, unlike SWE-Together with a single opencode binding. Scale AI is a data and model evaluation vendor that promotes its own benchmarks (The two benches out of the four that were recruited were from Scale AI.)

If we sum up the results on two articles at once, then we have signals from two angles. Meta measures how many user adjustments the agent needs before the result (strong) Scale measured how much capacity is lost when demands come in pieces (even the strong, half)

Practically this means: 1When choosing a model for real work with people, look at the multi-turn gap, not just the autonomous resolve rate. 2️ Of the two main modes of failure, technical errors are a question to the model, but forgotten requirements are treated by the process - a file with a plan, an explicit list of requirements, a commit and a review for each iteration. This is no longer a question of the benchmark, but of your binding.

#AI #AI4SDLC #Engineering #Agents #Evals #Research