AGENTS.md: Does a Context File Help the Agent?

A coding agent receives the context file alongside the task. The file often describes the repository, lists verification commands, and states the rules a team applies to changes. The Evaluating AGENTS.md study tests four model-and-harness combinations. Codex and Qwen Code receive AGENTS.md, while Claude Code receives CLAUDE.md. Each agent uses its own /init command to produce the generated files.

SWE-bench Lite provides 300 tasks from established Python projects. The comparison there is between runs with a generated file and runs without one. CTXbench adds a third condition with instructions committed by developers. It contains 138 tasks from 12 repositories with existing context files. Tasks within a repository share the same developer-written file, so the number of tasks says more about the variety of fixes than it does about the variety of project instructions.

Building CTXbench meant reconstructing the environment, specifying each task, and writing tests that failed before the fix and passed afterwards. The researchers used an agent for this work. Reviewers relaxed the tests for 25 of the 138 tasks because they were too closely fitted to the original patch and could reject other valid fixes. The host draws attention to what that means for a team’s own evaluation. A test tied closely to a reference patch can confuse resemblance to that patch with compliance with the requirements.

Generated files increased average run costs by 20% on SWE-bench Lite and 23% on CTXbench. The researchers found no statistically significant change in task success relative to runs without a file. On CTXbench, developer-written files outperformed generated ones. Their advantage over the no-file condition did not reach statistical significance. The host points out how the third version of the preprint qualifies the claims made in the first.

The traces help explain where the resources go. Agents use uv more often when instructed to use it. Testing guidance produces extra test runs, and guidance on exploration leads to more searches and reads. These are signs that agents follow the instructions. To assess the value of an overview, the researchers count steps to the first interaction with a file changed by the original pull request. No agent gets there appreciably faster. GPT-5.1 mini sometimes searches for and rereads the context file it has already been given.

Generated overviews often repeat the README, documentation, and examples. In an additional experiment, the researchers generate the context file and then remove those other materials. The retained file helps solve tasks in that setting. The value of the text depends on which facts the agent can get elsewhere.

The researchers also remove individual sections from generated files. Dropping guidance on tests or tools can lower costs in the additional GPT-5.2 runs, with little evidence of a change in task success from removing any one section. The host relates this to the purpose of an instruction. If a check matters to the team, measure its value on tasks where it can catch a consequential error.

The host applies the same reasoning to the issue description. A report that an export sometimes crashes can become a detailed task with scenarios to handle, a required order of operations, and a return type. Once the prompt includes those details, there is less for the context file to contribute to that run. The researchers show that clarifying SWE-bench Lite tasks improves the GPT-5.2 score. The host treats this as a possible limitation of the CTXbench design, where agents helped draft tasks and tests that the researchers then reviewed.

A working repository may need instructions for practices that are hard to infer from code. Examples in the episode include updating dependencies, ordering migrations, and editing the generator when generated output needs to change. For an export, a rule could name the verification command, point to the contract, and tell the agent to ask for the expected format when it is missing. Detailed guidance can live in a document the agent opens when relevant.

The host proposes three conditions: the original task without the file, the same task with the file, and a more explicit task with the relevant constraints in the request. The first pair measures what the file changes. The third shows what happens when the request makes the same information explicit. A team can record cost and task success alongside architectural conformance, security, and reliability. The study checks functional outcomes through tests. Properties the team cares about in the implementation need checks of their own.

The Q&A turns to Chad Fowler’s Regenerative Software. If an implementation can be rebuilt, the team needs a record of the intended behaviour, checks that demonstrate it, and the reasoning behind decisions. The host describes a drill in which a component is recreated from its requirements and tested against the behavioural checks. As the system evolves, the supporting material needs maintenance too, so obsolete rules and checks do not accumulate.

Repeated production issues can reveal where the instructions need work. The host describes asking an agent to group a month of errors and investigate their causes, then reviewing whether the proposed fixes fit the system. The findings can become a rule in AGENTS.md, a procedure in a skill, or an automated check. The discussion closes with a further way to retain experience. An agent could examine past sessions to build up knowledge of the project, giving the team a source from which to select durable rules for its context file.