SWE-Together: How to measure coding agents in dialogue rather than in one go (Category AI4SDLC)
I read the latest article.SWE-Together: Evaluating Coding Agents in Interactive User Sessions"from the team of the banned in Russia company "Meta" (arXiv, 29 June 2026). The authors hit the weak spot of almost all benchmarks of coding agents: they are static. The agent receives a full statement of the task immediately and is evaluated by the final code. And the real work with the agent is arranged differently - a dialogue where the user reveals the intention gradually, clarifies the requirements and corrects errors along the way. In general, the authors wanted to test the hypothesis that a strong model needs less user intervention. (||And that's really true.||)
In general, problems with benches are classic: one is saturated and cease to distinguish between top models. (see my analysis performances by SWE-rebench). Also, benches don’t always measure exactly how engineers interact with them. Actually, SWE-Together (leaderboard, github) He's just trying to get away from the one-way show. (single-turn) dialogue (multi-turn)It’s more like working with real agents.
Interesting data collection procedure
- It was at the entrance. 11,260 Real user sessions with coding agents (four open datasets with Hugging Face: DataClaw, Pi-staging, Hyperswitch, SWE-chat)
- We got it out. 109 reproducible tasks of the repository level, which gives a conversion to 0.97% In the middle, there were hard filters: a restoreable state of the repository, a clear target, a verifiable outcome, and the final change must be written mostly by the agent, not the person. Each task at the output is a sandbox with a fixed commit, environment and verification.
The most interesting engineering part of this bench is the user simulator. It is necessary because the initial session cannot be played verbatim: the new agent will go the other way, and the replicas of the original user will lose meaning. Therefore, the LLM simulator is anchored to the intentions from the initial session, but responds to the live trajectory of the agent being evaluated: after each move, it makes one decision - to remain silent, ask a question, redirect the agent, add a requirement or ask to check the external artifact. In a blind test, annotators were unable to distinguish the simulator from live users: 46Percentage of "taking for a person" at a random level 50%, the difference is statistically insignificant (passed the modified Turing test.).
Evaluation is also done with an eye on a known disease: fixed tests distort the score in both directions - narrow cling to random details of implementation, wide require behavior that no one asked for. Here, the final state of the repository is evaluated by an agent judge based on a pre-recorded set of criteria. (frozen rubric)Weighted behavioral goals are formed once, offline, before and independently of any candidates, and then applied equally to all models. Behavioral completeness is evaluated, not similarity to the original patch.
At the output, we have another metric User Correction, which shows how many times the user had to explicitly correct the agent. (plus soft nudges with weight 0.2) On the way to results. Two agents with the same final score can cost a user very different effort.
Results on seven models in general binding (harness) opencode.
- Claude Opus 4.8 Leading: pass@1 63%, judge's average score 0.801 And a minimum of adjustments. 1.38 per session, but also the largest expenditure output+reasoning tokens (74k on task)
- GPT-5.5 second-best judge (0.763) and the most economical: 29.9k tokens and 10.7 minute The ability and number of adjustments are almost linear: the Pearson correlation.0.92". As a result, the hypothesis that a strong model needs less user interventions has received quantitative confirmation - within this benchmark. Funny detail: models of the Meta in the table is not.
In general, the authors of the benchmark were able to measure the tax on the verification of the results of the agent’s work. (Using the User Correction Metric). AI has made it cheaper to produce code, but not trust in change, and “how many times a person had to send an agent” is a good estimate of that trust. It practically means two things. 1Evaluate agents is worth a couple of metrics - success plus the cost of taxiing, not one pass@1 2Your own sessions with agents are ready-made raw materials for such quality checks (evals): Pipeline "session → task" in the article is described in detail, the code is open.
P.S. Scale AI's adjacent whitepaper "SWE-Interact", released on the same day as SWE-Together, is reviewed in the journal Science. next post.
#AI #AI4SDLC #Engineering #Agents #Evals #Research