3 ms·

An Empirical Study of Harness Design for Coding Agents

29 points by wek 15d ago

Chloi 15d ago
[dead]
gps372 15d ago
Haven't gone through full PDF as its very detailed, few things have resonated with me so far. Basically if a Car A is performing better (be it speed, milage or in general sense) than Car B, then it is not necessarily because its engine. It could be because of better tires, better gearbox, lighter body, better usability of features, etc. You can implement an AI feature (like AI for BI) in different ways even with the same model - via ReAct-loop, or plan-and-execute, or hybrid. You can make it stateless, stateful, RAG-based, etc. depending upon whether you want to prioritize result accuracy or depth of analysis. You can use LLM to generate either intent (requires lesser reasoning) or the queries itself (requires much more capable model). Your harness can adapt to the underlying model's native capabilities, or can make up for its absence, e.g. query generation in above example requires your model to have MOE capabilities but intent generation wouldn't.
- yieldcrv 15d ago
  so then what I’m really interested in are benchmarks of OSS models vs closed source frontier models, using Claude Code as a harness
- BIGFOOT_EXISTS 15d ago
  Love your analogy. Cars performance is hugely dependent on use case and overall setup (suspension, engine (NA, turbo, supercharger, etc), coef of drag, etc). Maps well onto the LLM surface.
  - jasonwatkinspdx 15d ago
    Anyone that's into auto racing as a hobby will tell you the engine is the last thing you should modify unless your whole goal is to get into the guts of engine mechanics. But if you just want to improve lap times it's way less of a priority than brakes, tires, suspension setup, cooling to keep everything happy, etc.
    - bluefirebrand 15d ago
      I'm not a car guy but would it be fair to see that your maximum possible performance is bounded by the engine, but the average performance gains are from tuning the rest of the vehicle? Kind of like if your RAM has to constantly page, you would see much more improvement from upgrading your RAM than overclocking your CPU?
embedding-shape 15d ago
The conclusions: > Planning improves success at additional cost for weaker models but mainly reduces cost, with small decreases in success rate, for stronger models. > Predefined tools raise success rates for models with weak bash control, whereas bash-only yields higher success at lower cost for bash-capable models, most clearly on shell-centric task types. > context management extends execution trajectories without substantially altering agent behavior and is most beneficial under tight context budgets > planning sustains the trajectories of models that abandon tasks too early and trims repeated verification in models that verify too long > structured tools support models with limited shell proficiency, while bash-only enables capable models to combine multiple code modifications in a single tool call Seems fairly intuitive to me, based on feeling. But also fairly kind of obvious; bash-only tooling has higher success for bash-capable models, compared to using predefined tools for models that aren't good at bash? Yeah... They all seem a bit "duh" to me. The final piece of the conclusion is agreeable regardless of how they arrived at it though: > Harness design is thus a conditional systems problem in which each component should be selected for the target model, task type, and resource budget rather than adopted as a default. I think lots of people treat the harness/model/prompts combo as interchangeable, but in my experience the quality and efficiently depends heavily on the combo of the harness/model, and using the harness + model made by the same lab, has vastly better experience compared to more "general purpose" (for the lack of a better term) harnesses. Most likely because they use their own traces when training future model iterations.
- 0xbadcafebee 15d ago
  It might be 'duh' but it means we need a formal list of what each model is good at, and to pick or change harnesses to closer fit the model. Like an llm recipe book. Not just for remote models, but also local ones where how you run the model is critical too.
- zrail 15d ago
  "Everybody knows foul air causes sickness." "Duh, of course Mars has canals." Testing the "obvious", "duh" things is incredibly valuable science. It provides a more solid foundation on which to build because it reduces the assumption space.
- 15d ago
themgt 15d ago
Todo/task-tracking tools (TaskCreate/Get/Update/List, TodoWrite) are no longer available on Opus 4.8, Sonnet 5, Fable 5, Mythos 5, and newer models; set CLAUDE_CODE_ENABLE_TODO_TOOLS=1 to bring them back" Anthropic appears to agree frontier models don't need in-session planning tools. https://github.com/anthropics/claude-code/issues/80487 https://github.com/anthropics/claude-code/issues/80487
vblanco 15d ago
This is done on Nemotron models + mistral, so its not very relevant to the current frontier of cheap chinese models + big models from Claude/GPT. Big miss not having qwen or deepseek in this research.
- dsiegel2275 15d ago
  The focus of the study was the different harness approaches and how they scale across model sizes. The fact that they used any particular set of models is irrelevant.
  - shermantanktop 15d ago
    Agree. Harnesses are effective because they interact with the underlying model effectively. If the latest models were fundamentally different, excluding them would be a miss. But I don’t think they are, at least not in ways that would affect these observations.
    - svachalek 15d ago
      I think the confounding issue is that by now, millions of sessions of Claude Code and Codex are now in the training set for these models. So they have been trained to work the way these harnesses are configured, and at least in the case of Claude Code the harness itself is greatly stripped down because the model has absorbed it.
  - hiddencost 15d ago
    Nope. Sorry. Not how this works.
    - bjelkeman-again 15d ago
      How does it work then?
      - belowavgiq 15d ago
spncai 15d ago
[flagged]
corbinvachal 15d ago
[flagged]
agentdev001 15d ago
As far as I can tell, the paper says "bash capable", without ever describing what that means. How would one know whether a given model is "bash capable" or not? I would have to imagine, that Luna would very much fall into the camp of "bash capable". At which point- it seems to me that adding any tools beyond just Bash requires some rigorous testing and verification that value is being added.
- kridsdale1 15d ago
  I think it just means “is there some parser program that consumes the LLM token stream and spawns shell processes with the detected command strings”.
  - agentdev001 15d ago
    Im not sure how this could make sense, considering the content of the paper.
- everforward 15d ago
  I think it’s sort of self-defined. If a model is able to use bash well enough to not need specific tools. The research seems to agree with you, though. The paper calls out that for “bash capable” models, adding tools to do things bash can already do doesn’t improve performance. Vaguely the same result as RAG. Unless you’re in specific domains, you won’t beat handing the agent a shell and grep.
  - agentdev001 15d ago
    > Vaguely the same result as RAG. Unless you’re in specific domains, you won’t beat handing the agent a shell and grep. This has been my conclusion as well, and I'm doing my best to try and back this up quantitatively. In a perfect world, I could smite all of the internal MCP servers in my Corp environment and replace them with REST/GraphQL. No one using any of these servers is hitting them with models that would perform worse orchestrating with bash- and some of these folks are running harnesses with awful MCP clients.
  - 15d ago
vegadw 15d ago
Awesome work! I tried to get my lawyer-mom switched to Linux a bit ago, and she loved it in generally, but none of the Office competitors had good enough compat to work. The only other thing keeping her on Windows is Adobe PDF pro, since it can do OCR where, when you edit it, it reflow the text in a font that matches the scanned in one to look like the original. (I also got weird "This feels like it enables fraud vibes" from this, but, no, turns out it's a totally common workflow for lawyers to need to do this ... I hate it.) Cool to see 1/2 of the problems keeping her on Windows solved.
- lelanthran 15d ago
  > but none of the Office competitors had good enough compat to work. Don't worry about that - all the agents/harnesses/tools that produce .doc or .docx files do so with LibreOffice now. In a short time people would rather tell their agent "produce this file" and get one that is 99.9% compatible with Word than actually open Word and use the builtin copilot.
lieret 15d ago
Cool study, we definitely need more principled studies on the role of harnesses. I'd also say that there aren't too many benchmarks where the more complicated harnesses consistently outperform extremely simple agents. But I'm also biased, because I wrote https://github.com/swe-agent/mini-swe-agent/ https://github.com/swe-agent/mini-swe-agent/ , which is probably the most minimal agent out there (it started as just 100 lines, all included), and it's used in a lot of benchmarks like DeepSWE, terminalbench, programbench (seems like it's still top of the ranking for TB3, but wasn't evaluated with the best models on TB4).
- Systemerror7A69 15d ago
  I'd really love to see more studies about effectiveness of AI in general. As in, what works best and how to use it and such. Because I feel that the technology and space is - so - hyped and fast moving that a lot of cultish feeling rituals seem to pop up, none of which are backed by evidence. Anthropic openly recommend giving the agents.md file an architectural overview of the code, and the one time this was studied they found the opposite - that the agents.md file is best for concrete commands about how to build stuff and such, and - not - huge overviews. This was, and still is, the official recommendation from Anthropic as far as I can tell. And then there are the benchmarks, how feel vague and not concrete, and everyone kind of knows they're not the best cuz you can't just assign these tools one fixed number ( for multiple reasons ), but everyone still looks at them and compares them. People share skills and superpowers and plugins and mcps and very, very few of them have and kind of proof they do much at all. It all feels a bit weird to me, and I've been on the lookout for exactly these kinds of studies more lately, because I think having this research, even if not done on the exact newest models or not the exact, newest thing, are still - vastly - superior to the alternative.
  - DanielHB 15d ago
    > People share skills and superpowers and plugins and mcps and very, very few of them have and kind of proof they do much at all. At least 3 times last month I was asked to review a change in a .md file used by agents. And I am like: "yeah I guess it makes sense?" It feels we need to write unit tests for this stuff, but even how to do so in reasonable time and complexity seems difficult.
Aeroi 15d ago
nice work and paper, seeing more harness benchmarks emerge and we definitely need more. I ran mouse on the Frontier Harness benchmark and scored the highest pass rate, however I'm not convinced the results there actually translate to meaning the "best" harness in practice. Always looking for more harness evals, although I'm going broke running them across all these models and tasks.
mohd_rafay 15d ago
[flagged]
Jzuckerman 15d ago
[flagged]
rahulmax 15d ago
Quite inline with what I had found with my Claude code sessions over the last year. I wrote about this a few months ago. https://rahulmax.com/notes/how-i-keep-the-ai-bill-down/ https://rahulmax.com/notes/how-i-keep-the-ai-bill-down/ In their case, context management pays off more the tighter your window. Their gap between managing and not managing is 35.7 points of success rate at 32k and 2.7 points at 128k. My version of that was a rule I stick to, as much as I can. I checkpoint a session at about 25-30% of the window, write the state out to a PROGRESS.md and a JSON file of the requirements, and start fresh. This restart costs me 30 seconds, since a bloated session doesn't get any cheaper the longer you stay in it. Also worth knowing that the models are Nemotron-3 and Mistral-Medium, not the frontier models most people here are paying for.
- arcanemachiner 15d ago
  I've been pushing the context window well into the 400-600k+ token range lately (mostly Opus 5). I prefer not to since I'm aware of context collapse, but I've been leaning that way lately. Lately, I've been finding that the game of telephone of handoffs causes more mistakes than just letting the session run longer. Not too mention the wasted time waiting for agents to poke around as they bootstrap a new session from the handoff. Of course, I still have handoffs for the large-scale plan being accomplished, but I've been having better results letting an agent finishing what it starts. The game of telephone is a painful waste of time when it goes wrong, which is far too often for me.
  - mmykola87 13d ago
    [dead]
- bunderbunder 15d ago
  And what you describe doing is also very in line with Cursor's published recommendations. Though of course every single element of their application's actual UX is pushing hard in the opposite direction. The cynical part of me can't help but notice that there's a bit of a conundrum here: using coding agents more effectively seems to involve working pretty hard at giving the agent vendor less money.
practicalsystem 15d ago
definitely sending this to claude to take lessons from it and audit my harness, thanks for sharing
ellahayesus 15d ago
[flagged]
deleted 15d ago
[deleted]
goun7 15d ago
[flagged]
hosamsh 15d ago
Planning and tooling are suitably designed for execution. That's why they fail to decide on the resulting coding accuracy. If the product is user-facing (i.e. all products), a blind verification harness that is specifically designed to behave like a real user should be the decider.
kaufmann 15d ago
Something that totally confuses me is the use of the term “harness.” Is it the harness that enables the LLM to use tools and implement plans, or is it the short leash that, through many guardrails, ensures that the LLM follows the desired path? I see both meanings used. The former is still plausible, but I see many people using the second interpretation.
wesleysimplicio 15d ago
[flagged]
blinkbox 15d ago
[flagged]

29 points by wek 15d ago

Chloi 15d ago

[dead]

Haven't gone through full PDF as its very detailed, few things have resonated with me so far. Basically if a Car A is performing better (be it speed, milage or in general sense) than Car B, then it is not necessarily because its engine. It could be because of better tires, better gearbox, lighter body, better usability of features, etc. You can implement an AI feature (like AI for BI) in different ways even with the same model - via ReAct-loop, or plan-and-execute, or hybrid. You can make it stateless, stateful, RAG-based, etc. depending upon whether you want to prioritize result accuracy or depth of analysis. You can use LLM to generate either intent (requires lesser reasoning) or the queries itself (requires much more capable model). Your harness can adapt to the underlying model's native capabilities, or can make up for its absence, e.g. query generation in above example requires your model to have MOE capabilities but intent generation wouldn't.

yieldcrv 15d ago
so then what I’m really interested in are benchmarks of OSS models vs closed source frontier models, using Claude Code as a harness
BIGFOOT_EXISTS 15d ago
Love your analogy. Cars performance is hugely dependent on use case and overall setup (suspension, engine (NA, turbo, supercharger, etc), coef of drag, etc). Maps well onto the LLM surface.
- jasonwatkinspdx 15d ago
  Anyone that's into auto racing as a hobby will tell you the engine is the last thing you should modify unless your whole goal is to get into the guts of engine mechanics. But if you just want to improve lap times it's way less of a priority than brakes, tires, suspension setup, cooling to keep everything happy, etc.
  - bluefirebrand 15d ago
    I'm not a car guy but would it be fair to see that your maximum possible performance is bounded by the engine, but the average performance gains are from tuning the rest of the vehicle? Kind of like if your RAM has to constantly page, you would see much more improvement from upgrading your RAM than overclocking your CPU?

embedding-shape 15d ago

The conclusions: > Planning improves success at additional cost for weaker models but mainly reduces cost, with small decreases in success rate, for stronger models. > Predefined tools raise success rates for models with weak bash control, whereas bash-only yields higher success at lower cost for bash-capable models, most clearly on shell-centric task types. > context management extends execution trajectories without substantially altering agent behavior and is most beneficial under tight context budgets > planning sustains the trajectories of models that abandon tasks too early and trims repeated verification in models that verify too long > structured tools support models with limited shell proficiency, while bash-only enables capable models to combine multiple code modifications in a single tool call Seems fairly intuitive to me, based on feeling. But also fairly kind of obvious; bash-only tooling has higher success for bash-capable models, compared to using predefined tools for models that aren't good at bash? Yeah... They all seem a bit "duh" to me. The final piece of the conclusion is agreeable regardless of how they arrived at it though: > Harness design is thus a conditional systems problem in which each component should be selected for the target model, task type, and resource budget rather than adopted as a default. I think lots of people treat the harness/model/prompts combo as interchangeable, but in my experience the quality and efficiently depends heavily on the combo of the harness/model, and using the harness + model made by the same lab, has vastly better experience compared to more "general purpose" (for the lack of a better term) harnesses. Most likely because they use their own traces when training future model iterations.

0xbadcafebee 15d ago
It might be 'duh' but it means we need a formal list of what each model is good at, and to pick or change harnesses to closer fit the model. Like an llm recipe book. Not just for remote models, but also local ones where how you run the model is critical too.
zrail 15d ago
"Everybody knows foul air causes sickness." "Duh, of course Mars has canals." Testing the "obvious", "duh" things is incredibly valuable science. It provides a more solid foundation on which to build because it reduces the assumption space.
15d ago

themgt 15d ago

Todo/task-tracking tools (TaskCreate/Get/Update/List, TodoWrite) are no longer available on Opus 4.8, Sonnet 5, Fable 5, Mythos 5, and newer models; set CLAUDE_CODE_ENABLE_TODO_TOOLS=1 to bring them back" Anthropic appears to agree frontier models don't need in-session planning tools. https://github.com/anthropics/claude-code/issues/80487 https://github.com/anthropics/claude-code/issues/80487

vblanco 15d ago

This is done on Nemotron models + mistral, so its not very relevant to the current frontier of cheap chinese models + big models from Claude/GPT. Big miss not having qwen or deepseek in this research.

dsiegel2275 15d ago
The focus of the study was the different harness approaches and how they scale across model sizes. The fact that they used any particular set of models is irrelevant.
- shermantanktop 15d ago
  Agree. Harnesses are effective because they interact with the underlying model effectively. If the latest models were fundamentally different, excluding them would be a miss. But I don’t think they are, at least not in ways that would affect these observations.
  - svachalek 15d ago
    I think the confounding issue is that by now, millions of sessions of Claude Code and Codex are now in the training set for these models. So they have been trained to work the way these harnesses are configured, and at least in the case of Claude Code the harness itself is greatly stripped down because the model has absorbed it.
- hiddencost 15d ago
  Nope. Sorry. Not how this works.
  - bjelkeman-again 15d ago
    How does it work then?
    - belowavgiq 15d ago

spncai 15d ago

[flagged]

corbinvachal 15d ago

[flagged]

agentdev001 15d ago

As far as I can tell, the paper says "bash capable", without ever describing what that means. How would one know whether a given model is "bash capable" or not? I would have to imagine, that Luna would very much fall into the camp of "bash capable". At which point- it seems to me that adding any tools beyond just Bash requires some rigorous testing and verification that value is being added.

kridsdale1 15d ago
I think it just means “is there some parser program that consumes the LLM token stream and spawns shell processes with the detected command strings”.
- agentdev001 15d ago
  Im not sure how this could make sense, considering the content of the paper.
everforward 15d ago
I think it’s sort of self-defined. If a model is able to use bash well enough to not need specific tools. The research seems to agree with you, though. The paper calls out that for “bash capable” models, adding tools to do things bash can already do doesn’t improve performance. Vaguely the same result as RAG. Unless you’re in specific domains, you won’t beat handing the agent a shell and grep.
- agentdev001 15d ago
  > Vaguely the same result as RAG. Unless you’re in specific domains, you won’t beat handing the agent a shell and grep. This has been my conclusion as well, and I'm doing my best to try and back this up quantitatively. In a perfect world, I could smite all of the internal MCP servers in my Corp environment and replace them with REST/GraphQL. No one using any of these servers is hitting them with models that would perform worse orchestrating with bash- and some of these folks are running harnesses with awful MCP clients.
- 15d ago

vegadw 15d ago

Awesome work! I tried to get my lawyer-mom switched to Linux a bit ago, and she loved it in generally, but none of the Office competitors had good enough compat to work. The only other thing keeping her on Windows is Adobe PDF pro, since it can do OCR where, when you edit it, it reflow the text in a font that matches the scanned in one to look like the original. (I also got weird "This feels like it enables fraud vibes" from this, but, no, turns out it's a totally common workflow for lawyers to need to do this ... I hate it.) Cool to see 1/2 of the problems keeping her on Windows solved.

lelanthran 15d ago
> but none of the Office competitors had good enough compat to work. Don't worry about that - all the agents/harnesses/tools that produce .doc or .docx files do so with LibreOffice now. In a short time people would rather tell their agent "produce this file" and get one that is 99.9% compatible with Word than actually open Word and use the builtin copilot.

lieret 15d ago

Cool study, we definitely need more principled studies on the role of harnesses. I'd also say that there aren't too many benchmarks where the more complicated harnesses consistently outperform extremely simple agents. But I'm also biased, because I wrote https://github.com/swe-agent/mini-swe-agent/ https://github.com/swe-agent/mini-swe-agent/ , which is probably the most minimal agent out there (it started as just 100 lines, all included), and it's used in a lot of benchmarks like DeepSWE, terminalbench, programbench (seems like it's still top of the ranking for TB3, but wasn't evaluated with the best models on TB4).

Systemerror7A69 15d ago
I'd really love to see more studies about effectiveness of AI in general. As in, what works best and how to use it and such. Because I feel that the technology and space is - so - hyped and fast moving that a lot of cultish feeling rituals seem to pop up, none of which are backed by evidence. Anthropic openly recommend giving the agents.md file an architectural overview of the code, and the one time this was studied they found the opposite - that the agents.md file is best for concrete commands about how to build stuff and such, and - not - huge overviews. This was, and still is, the official recommendation from Anthropic as far as I can tell. And then there are the benchmarks, how feel vague and not concrete, and everyone kind of knows they're not the best cuz you can't just assign these tools one fixed number ( for multiple reasons ), but everyone still looks at them and compares them. People share skills and superpowers and plugins and mcps and very, very few of them have and kind of proof they do much at all. It all feels a bit weird to me, and I've been on the lookout for exactly these kinds of studies more lately, because I think having this research, even if not done on the exact newest models or not the exact, newest thing, are still - vastly - superior to the alternative.
- DanielHB 15d ago
  > People share skills and superpowers and plugins and mcps and very, very few of them have and kind of proof they do much at all. At least 3 times last month I was asked to review a change in a .md file used by agents. And I am like: "yeah I guess it makes sense?" It feels we need to write unit tests for this stuff, but even how to do so in reasonable time and complexity seems difficult.

Aeroi 15d ago

nice work and paper, seeing more harness benchmarks emerge and we definitely need more. I ran mouse on the Frontier Harness benchmark and scored the highest pass rate, however I'm not convinced the results there actually translate to meaning the "best" harness in practice. Always looking for more harness evals, although I'm going broke running them across all these models and tasks.

mohd_rafay 15d ago

[flagged]

Jzuckerman 15d ago

[flagged]

rahulmax 15d ago

Quite inline with what I had found with my Claude code sessions over the last year. I wrote about this a few months ago. https://rahulmax.com/notes/how-i-keep-the-ai-bill-down/ https://rahulmax.com/notes/how-i-keep-the-ai-bill-down/ In their case, context management pays off more the tighter your window. Their gap between managing and not managing is 35.7 points of success rate at 32k and 2.7 points at 128k. My version of that was a rule I stick to, as much as I can. I checkpoint a session at about 25-30% of the window, write the state out to a PROGRESS.md and a JSON file of the requirements, and start fresh. This restart costs me 30 seconds, since a bloated session doesn't get any cheaper the longer you stay in it. Also worth knowing that the models are Nemotron-3 and Mistral-Medium, not the frontier models most people here are paying for.

arcanemachiner 15d ago
I've been pushing the context window well into the 400-600k+ token range lately (mostly Opus 5). I prefer not to since I'm aware of context collapse, but I've been leaning that way lately. Lately, I've been finding that the game of telephone of handoffs causes more mistakes than just letting the session run longer. Not too mention the wasted time waiting for agents to poke around as they bootstrap a new session from the handoff. Of course, I still have handoffs for the large-scale plan being accomplished, but I've been having better results letting an agent finishing what it starts. The game of telephone is a painful waste of time when it goes wrong, which is far too often for me.
- mmykola87 13d ago
  [dead]
bunderbunder 15d ago
And what you describe doing is also very in line with Cursor's published recommendations. Though of course every single element of their application's actual UX is pushing hard in the opposite direction. The cynical part of me can't help but notice that there's a bit of a conundrum here: using coding agents more effectively seems to involve working pretty hard at giving the agent vendor less money.

practicalsystem 15d ago

definitely sending this to claude to take lessons from it and audit my harness, thanks for sharing

ellahayesus 15d ago

[flagged]

deleted 15d ago

[deleted]

goun7 15d ago

[flagged]

hosamsh 15d ago

Planning and tooling are suitably designed for execution. That's why they fail to decide on the resulting coding accuracy. If the product is user-facing (i.e. all products), a blind verification harness that is specifically designed to behave like a real user should be the decider.

kaufmann 15d ago

Something that totally confuses me is the use of the term “harness.” Is it the harness that enables the LLM to use tools and implement plans, or is it the short leash that, through many guardrails, ensures that the LLM follows the desired path? I see both meanings used. The former is still plausible, but I see many people using the second interpretation.

wesleysimplicio 15d ago

[flagged]

blinkbox 15d ago

[flagged]