A new study from ETH Zurich challenges a widespread practice in software development: the use of context files like Agents.md to adapt AI coding agents to repositories. Although developers routinely deploy these files to make agents more productive and reduce errors, the research now shows that costs rise significantly without any measurable benefit.
Key Takeaways
- Context files like Agents.md are established practice, recommended by developers themselves, but their effectiveness had never been scientifically validated
- The ETH study examined four major coding agents: Claude Code (Anthropic, Sonnet-4.5), Codex (OpenAI, GPT-5.2 and GPT-5.1 Mini), and Qwen Code (Alibaba, Qwen3-30B-Coder)
- CTXbench benchmark with 138 real GitHub issues from twelve niche projects shows: context files increase costs without improving success rates
- Results available on Arxiv and advise against using this practice
How the Researchers Approached It
The team from ETH Zurich and Swiss startup Logicstar.ai developed two evaluation approaches. First, they used SWE-Bench, an established benchmark that measures how well AI coding agents solve real software problems in large, well-known projects.
Second, the researchers created a new benchmark called CTXbench. This focuses specifically on smaller, lesser-known repositories – the type typically developed by mid-market companies or startups. For CTXbench, they selected 138 real GitHub issues from twelve niche projects that already had developer-written context files. This allowed them to test the practice under realistic conditions.
The Surprising Findings
The study reveals a clear pattern: while context files significantly increase token costs for AI models, there is no corresponding improvement in solution quality. In other words: developers pay more but don't get better results.
This is particularly relevant for teams working with paid APIs like OpenAI or Anthropic. Every additional token in the input costs money – and if those tokens deliver no measurable value, the investment is wasted.
What This Means for Organizations
These findings should prompt development teams worldwide to reconsider their approach. Many organizations have adopted Agents.md and similar context files as best practice without questioning their return on investment. The ETH study suggests this practice should be re-examined during KI cost optimization – especially when AI budgets are under pressure.
The question remains: are there alternative ways to instruct coding agents efficiently without driving costs up? For teams already invested in context files, this study could be a catalyst to validate their actual effectiveness in their own projects.
Sources
Editorially owned by Ideal Syka. Sources and method: Newsroom & method. Tips and corrections: ai@i6eal.de.
