Research Insights #47 Materials: Does AGENTS.md Help the Agent? (Series #AI4SDLC)

On November 10, 2026, my solo Research Insights Made Simple #47 episode came out. It covers “Evaluating AGENTS.md (v3)”, which I wrote about in an earlier post. Suppose we put which tests to run in a file. The agent runs them more often. Does it solve more tasks? In the episode I looked at an experiment from ETH Zurich and LogicStar.ai where those two outcomes diverged.

In the episode

🔸 How benchmark cost and success change Generated instructions raised average run cost by 20% on SWE-bench Lite and 23% on CTXbench. The authors found no statistically significant increase in the share of tasks solved. On CTXbench, developer-written files worked better than generated ones. Their advantage over runs without a file wasn’t statistically confirmed.

🔸 Where the extra steps go If the instructions mention uv, the agent uses uv more often. If it is asked to run tests, it runs them more often. The authors tracked these actions separately and counted a task as solved based on tests of the patch’s behavior. If following the order of migrations matters to you, test that property separately.

🔸 What the file adds to what the agent can already read In an additional CTXbench experiment, researchers generated the file first and then removed documentation and examples from the repository. Under those conditions, the generated file helped solve tasks. A summary of the README becomes valuable when the README itself is no longer on hand.

🔸 What the task wording changes “Export sometimes fails” and “here’s how to reproduce the failure and what should happen” leave the agent with different amounts of guesswork. My hypothesis is that the more precise the task is, the less useful information the file may have left to add.

🔸 How to run your own experiment Pick one task. First give the agent the original prompt without the file, then the same prompt with the file. In the third version, move the relevant rules directly into the task. Compare cost, check whether the patch works, and see whether the rule itself was followed. For instance, did the agent fix the generator if the file in question is generated automatically?

Episode materials

— Episode page with timestamps and slides — Video: YouTube, VK Video — Audio: Podster, Apple Podcasts — Written summary

If you have a rule in AGENTS.md that came out of a specific mistake by an agent, what happened and how did you tell the rule helped?

#ResearchInsights #AI4SDLC #Agents #Evals #Engineering