Evaluating AGENTS.md: What Context Is Still Missing?

Evaluating AGENTS.md: What Context Is Still Missing? (Category #AI4SDLC)

The paper “Evaluating AGENTS.md” has a curious result: instructions gave agents more work, with no convincing increase in the number of tasks they solved. I read the paper and annotated it. I'll start with the experiment and then get to the questions it left me with.

Thibaud Gloaguen and his co-authors from ETH Zurich and LogicStar.ai tested how useful files such as AGENTS.md and CLAUDE.md are. This write-up covers v3 of the preprint, dated September 29, 2026. The authors used SWE-bench Lite, with 300 tasks, and built their own CTXBench: 138 tasks from 12 Python repositories with AGENTS.md files written by the repository developers. They then compared four model-and-agent combinations: Claude Code with Sonnet 4.5, Codex with GPT-5.2 and GPT-5.1 Mini, and Qwen Code with Qwen3-30B-Coder. There were three conditions: without AGENTS.md, with an AGENTS.md generated by Claude or Codex, and with an AGENTS.md written by the repository developers. The last condition was available only in CTXBench, which was built from repositories that had such a file. They measured success by running tests on the patch the agent wrote.

What they found

1️⃣ Auto-generation raised average cost by 20% on SWE-bench and 23% on CTXBench. Compared with the no-instructions condition, the changes in success rate were statistically insignificant: a small drop with auto-generated files and a small improvement with developer-written files. The developer-written files did, however, significantly outperform the generated ones on CTXBench.

2️⃣ Agents followed the instructions: they read and tested more and used the specified tools. Repository overviews did not help them reach the files they needed any faster. The authors recommend keeping specific requirements that aren't already in the documentation and testing the value of the instructions experimentally.

And now for my notes in the margins

What caught my attention was how much work agents had already done to prepare CTXBench. One filters PRs, another sets up the environment, the next writes a detailed task description from the PR and its completed patch, and another generates tests. There's also an LLM for analyzing the context files and the agents' actions, but that's part of measurement and doesn't itself add detail to the task description.

The step I find particularly interesting is “write the task from the finished solution.” The authors forbid disclosing the solution and check the result manually. But knowing the answer lets you pick out the important requirements, interfaces, and edge cases in advance. The agent gets no hint about the code, yet some of the work of figuring out the task has already been done.

Imagine the distance between “the export sometimes crashes” and “handle an empty list, preserve the order, and return this type.” It takes a fair bit of investigation to get from one to the other, even if nobody tells you how to fix the bug.

So my hypothesis is that the added value of AGENTS.md may shrink when the uncertainty has already been resolved while preparing the task. Especially if the file was also generated from the repository the agent can already access. What meaningful information is left for it to add?

The paper has some interesting checks. Rewriting SWE-bench tasks this way raises success rates by roughly 14–17 percentage points for the two models tested in the comparison without context files. When the rest of the documentation is removed, the generated instructions yield an average gain of 2.7 percentage points. So the information available at the outset matters.

Task preparation can't explain everything, though. Auto-generation didn't produce a significant improvement on the original SWE-bench either. My hypothesis about the mechanism remains a hypothesis.

Two more notes:
1) Everything was tested on Python, which seems to be the most common language in pretraining.
2) Passing tests doesn't cover all non-functional requirements. The value of rules about security, architecture, or maintainability needs to be evaluated separately.

To extend the experiment, I'd compare the original and clarified tasks with the same developer-written AGENTS.md. I'd also test whether it helps uncover requirements that nobody has turned into a finished specification yet.

#AI4SDLC #Research #Agents #Evals #Engineering