4 ms·

We really need better harness benchmarks. It seems there's no reliable source that benchmarks the main harnesses against all open source models. I also wish th

by nojs 9d ago

We really need better harness benchmarks. It seems there's no reliable source that benchmarks the main harnesses against all open source models. I also wish the discussion around Pi did not always use cost/token count as the metric. It's amazingly token efficient, but how does it stack up again opencode and others if you don't care about token count? My experience is that the harness is mainly polish preventing failed tool calls, bad edits, stuff like that, but doesn't make much difference to the overall "intelligence". But that opencode seems slightly more robust against stupid errors than out of the box Pi due to the additional context it forces through every thread.

sn0n 9d ago
A good benchmark would require a decent number of smaller scoped one off tasks to larger multi step refactors, and also one shot full project of simple to complex varieties. In addition to a series of “conversational” ambiguity filled one-liners.
leemysw 9d ago
[flagged]
calgoo 9d ago
I really disliked how opencode works IMO; the harness tries to do to much in my mind. Switching to Pi was a breath of fresh air for me, and I even use hax for some of my local needs where i dont want to have the giant pile of fertilizer that is NPM or PIP installed. The harness becomes more and more important, the smaller the model is as you need to offload context management as well as memory to the harness. The big models basically just need a bash prompt tooling and you let the model manage everything inside its own context.
- mobelkh 9d ago
  what does it do too much out if the box? afaik it only has the basic tools and two agents profiles, maybe lsp support is extra?
  - tacomagick 9d ago
    Moving from CC I can agree OpenCode seems very simple. I don't see what is too much about it honestly.
  - verdverm 9d ago
    look into the prompts, there are some choices in there disclaimer, I use opencode and have customized parts of it, and will do more, but it is a solid foundation and comes with more out of the box than pi pi is too minimal for me, I'd go back to my custom built harness if I wanted to be back at that level
    - esperent 9d ago
      > look into the prompts, there are some choices in there Such as?
      - verdverm 9d ago
        Instructing to use the `gh` CLI, not ideal for those of use using BBC at work They also have instructions about how to format certain output, which conflicts with the instructions we have in repo. I only discovered yesterday because we were wondering why the agent kept picking certain tools.
        esperent 9d ago
        What is BBC? Other than the tv network.
        verdverm 9d ago
        Bitbucket Cloud (Atlassian)
        deleted 9d ago
        [deleted]
sandeepkd 9d ago
Harness and benchmark for the harness feels like a chicken and egg problem. The harness is to optimize the interaction results with the models. Any benchmark for harness has to focus on the goals that the harness was trying to optimize for unless we are only focussing on generic harnesses. At this point when all the models have been trained on all available data with the similar algorithm, 1. either you get more data which is not feasible, 2. or get a better algorithm - a possibility , 3. or write a more targeted harness. Harnesses for legal, medicine and all are the ones which are getting focus for this reason. Writing benchmarks for these targeted harnesses would be a catching task
general_reveal 9d ago
[flagged]
- tomhow 8d ago
  > lizard satanists > Tell a horny monkey not to jerk off. Can you just not post garbage like this on HN. It's okay to criticize the big tech companies or general AI discussion here. Many do, it’s fine. But dreck like that only makes you seem unhinged and is the surest way to turn this place into the cesspool you say you're concerned about. Honestly. Some people have to read this stuff whether they want to or not. And to post this utter filth in a comment that's appealing for higher standards? Good grief. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
ebrahimisoheil 9d ago
[flagged]
tomrod 9d ago
I've been thinking about this a lot in the past few weeks - https://www.roderick.dev/writing/2026-08-28-obsessing-harnesses/ https://www.roderick.dev/writing/2026-08-28-obsessing-harnes... I don't think yet that a general benchmark will capture what is needed, but a gain/loss of function along with improvement / loss along the ability of that function seems to be a good path forward.
artdigital 9d ago
Very curious as well to see comparisons between Claude Code, Codex, Pi, Grok, OpenCode, Crush, etc Personally jumping around a lot to get a feeling for exactly that, and these days liking the Grok harness out of all of them the most
verdverm 9d ago
Coming Soon: https://artificialanalysis.ai/agents/coding-agents?coding-agents-performance-chart=swe-atlas-qna&coding-agents-harness-comparison-chart=harness-deep-swe#harness-comparison https://artificialanalysis.ai/agents/coding-agents?coding-ag... They used to have one chart with {model X harness} for a subset of combos, looks like that is getting an upgrade
jiaosdjf 7d ago
We really do need better benchmarks and for models too - Most people use a harness because of its subscription (most companies pay Anthropic) - All model benchmarks are biased and gamed, harness benchmarks are too few to matter - Everyone is just guessing, acting on sample sizes of 1 and trust me bro vibes

by nojs 9d ago

sn0n 9d ago

A good benchmark would require a decent number of smaller scoped one off tasks to larger multi step refactors, and also one shot full project of simple to complex varieties. In addition to a series of “conversational” ambiguity filled one-liners.

leemysw 9d ago

[flagged]

calgoo 9d ago

I really disliked how opencode works IMO; the harness tries to do to much in my mind. Switching to Pi was a breath of fresh air for me, and I even use hax for some of my local needs where i dont want to have the giant pile of fertilizer that is NPM or PIP installed. The harness becomes more and more important, the smaller the model is as you need to offload context management as well as memory to the harness. The big models basically just need a bash prompt tooling and you let the model manage everything inside its own context.

mobelkh 9d ago
what does it do too much out if the box? afaik it only has the basic tools and two agents profiles, maybe lsp support is extra?
- tacomagick 9d ago
  Moving from CC I can agree OpenCode seems very simple. I don't see what is too much about it honestly.
- verdverm 9d ago
  look into the prompts, there are some choices in there disclaimer, I use opencode and have customized parts of it, and will do more, but it is a solid foundation and comes with more out of the box than pi pi is too minimal for me, I'd go back to my custom built harness if I wanted to be back at that level
  - esperent 9d ago
    > look into the prompts, there are some choices in there Such as?
    - verdverm 9d ago
      Instructing to use the `gh` CLI, not ideal for those of use using BBC at work They also have instructions about how to format certain output, which conflicts with the instructions we have in repo. I only discovered yesterday because we were wondering why the agent kept picking certain tools.
      - esperent 9d ago
        What is BBC? Other than the tv network.
        verdverm 9d ago
        Bitbucket Cloud (Atlassian)
        deleted 9d ago
        [deleted]

sandeepkd 9d ago

Harness and benchmark for the harness feels like a chicken and egg problem. The harness is to optimize the interaction results with the models. Any benchmark for harness has to focus on the goals that the harness was trying to optimize for unless we are only focussing on generic harnesses. At this point when all the models have been trained on all available data with the similar algorithm, 1. either you get more data which is not feasible, 2. or get a better algorithm - a possibility , 3. or write a more targeted harness. Harnesses for legal, medicine and all are the ones which are getting focus for this reason. Writing benchmarks for these targeted harnesses would be a catching task

general_reveal 9d ago

[flagged]

tomhow 8d ago
> lizard satanists > Tell a horny monkey not to jerk off. Can you just not post garbage like this on HN. It's okay to criticize the big tech companies or general AI discussion here. Many do, it’s fine. But dreck like that only makes you seem unhinged and is the surest way to turn this place into the cesspool you say you're concerned about. Honestly. Some people have to read this stuff whether they want to or not. And to post this utter filth in a comment that's appealing for higher standards? Good grief. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html

ebrahimisoheil 9d ago

[flagged]

tomrod 9d ago

I've been thinking about this a lot in the past few weeks - https://www.roderick.dev/writing/2026-08-28-obsessing-harnesses/ https://www.roderick.dev/writing/2026-08-28-obsessing-harnes... I don't think yet that a general benchmark will capture what is needed, but a gain/loss of function along with improvement / loss along the ability of that function seems to be a good path forward.

artdigital 9d ago

Very curious as well to see comparisons between Claude Code, Codex, Pi, Grok, OpenCode, Crush, etc Personally jumping around a lot to get a feeling for exactly that, and these days liking the Grok harness out of all of them the most

verdverm 9d ago

Coming Soon: https://artificialanalysis.ai/agents/coding-agents?coding-agents-performance-chart=swe-atlas-qna&coding-agents-harness-comparison-chart=harness-deep-swe#harness-comparison https://artificialanalysis.ai/agents/coding-agents?coding-ag... They used to have one chart with {model X harness} for a subset of combos, looks like that is getting an upgrade

jiaosdjf 7d ago

We really do need better benchmarks and for models too - Most people use a harness because of its subscription (most companies pay Anthropic) - All model benchmarks are biased and gamed, harness benchmarks are too few to matter - Everyone is just guessing, acting on sample sizes of 1 and trust me bro vibes