AGENTS.md: does a context file help an agent?
Four agents, 300 + 138 tasks: the success rate barely moves while cost rises 19–23% on average
Slide contents
1. AGENTS.md: does a context file help an agent?
Four agents, 300 + 138 tasks: the success rate barely moves while cost rises 19–23% on average
2. Six stops around one file
What was tested, how, what came out and what to do
3. A short file isn't proven to help
In episode 35 AGENTS.md became a contents page; Giga Conf: “words shrink”
4. Context files spread before anyone tested them
A format since August 2025, /init in agents, 60,000+ repositories
5. A generated file: overview, tooling, tests
A developer file averages 641 words and 9.7 sections
6. Two setups answer different questions
SWE‑bench Lite for /init files; CTXbench adds developers' own files
7. From 5,694 pull requests to 138 tasks
Five steps: search, filter, setup, description, tests
8. Mostly niche projects instead of Django
From 17 tasks in openai-agents-python to 3 in graphiti
9. Three conditions across four agents
No file, /init file, developer file; one run each
10. Two points is about three tasks
Standard error ±3–4 pp, one run, a significance test
11. On SWE-bench generated files don't change accuracy
Minus 0.5 pp on average across four agents, p = 0.87
12. Developer files beat generated ones
Yet neither significantly beats no file: p = 0.37 and 0.21
13. Every context file raises the cost
Generated: +20% and +23%; developer files: about +19%
14. A repository overview barely shortens search
Not meaningfully fewer steps to the target file; more for GPT‑5.1 mini
15. Agents generally follow the file's rules
Mention uv and it gets 1.6 calls per task instead of almost none
16. With a file, OpenAI models reason longer
GPT‑5.2 spends up to 22% more reasoning tokens
17. Without docs, generated files help a little
With .md, examples and docs/ removed, generated files gain 2.7% on CTXbench
18. Neither length nor sections change accuracy
Without the testing section: 16–21% cheaper (our calculation)
19. The paper's conclusion grew more careful
February: “tend to reduce success”; from June: “does not generally improve”
20. Twelve files, only Python, one run
What the authors concede and what the host adds
21. Neighbour study: faster runs with the file
10 projects, 124 pull requests: median runtime −28.64%, output tokens −16.58%
22. Keep only what the README lacks
Non-standard rules, yes; overviews and retold docs, no
23. Context files are words you pay for
Generated files don't raise the success rate
Developer files beat generated ones
Any file costs 19–23% more on average
Agents follow rules, even needless ones
Without docs, generated files help a little
Write only what the agent can't learn itself