An Empirical Study of Harness Design for Coding Agents
The paper pushes toward model- and budget-aware harness design, but the discussion questions whether its results generalize beyond the tested models and whether simple bash-only or minimal agents already capture much of the benefit.
The brief
The paper studies coding-agent harness design by fixing the execution loop and varying planning, action space, and context management across four models on SWE-Bench Verified and Terminal-Bench 2.1, covering 176 matched settings. It reports that context management matters most under tight context budgets, rule-based elision before LLM summarization is the most efficient context strategy, planning shifts from an accuracy aid for weaker models to a cost saver for stronger models, and bash-only interfaces can be cheaper for bash-capable models while predefined tools help weaker bash users. Trajectory analysis links these effects to longer execution, different stopping points, and changed code-writing granularity.
- The study fixes the harness execution loop and varies three components: planning, action space, and context management.
- It evaluates four models on SWE-Bench Verified and Terminal-Bench 2.1 across 176 matched settings, including five context-management strategies and four context-window budgets.
- Context management becomes more valuable as the context-window budget tightens, mainly by preventing context-overflow failures.
- Staging rule-based elision before LLM-based summarization gives the strongest overall efficiency; making elided content recoverable adds machinery that models rarely use and yields no accuracy gain.
- Planning acts as an accuracy scaffold for weaker models but mainly reduces cost for stronger models, with little accuracy change.
- Predefined tools improve performance for models with weaker bash proficiency, while bash-capable models can use a bash-only interface effectively at substantially lower cost, especially on command-line-centric tasks.
In the discussion
Commenters largely treat the paper's conclusions as intuitive, while debating model coverage, the value of complex harnesses, and how to interpret the bash-only result.
Conclusions feel obvious but conditional harness design rings true
One commenter says the findings seem fairly intuitive and even obvious, but agrees with the paper's framing that harness design is conditional on model, task type, and resource budget. They add that harness/model/prompt combinations matter heavily and that using a harness and model from the same lab often produces a much better experience, possibly because labs train future models on their own traces.
Model selection limits relevance
A commenter argues the study is not very relevant to the current frontier because it uses Nemotron and Mistral models rather than cheap Chinese models or large Claude/GPT models, and calls the absence of Qwen or DeepSeek a big miss.
Model choice may not matter for the study's question
Another commenter counters that the study's focus is different harness approaches and how they scale across model sizes, so the particular set of models used is irrelevant.
Simple agents remain competitive
A commenter welcomes more principled harness studies but notes there are not many benchmarks where more complicated harnesses consistently outperform extremely simple agents. They point to mini-swe-agent, a minimal agent used in benchmarks such as DeepSWE, TerminalBench, and ProgramBench.
Bash-only result is read as a rebuke to MCP-style tooling
One commenter clarifies the bash finding as bash-capable models plus bash-only tools outperforming bash-capable models plus predefined tools. They argue this means MCP-style tooling helped only a little until models became good at bash and is now basically useless.