3 ms·

I for one think that harness development is perhaps the most interesting part at the moment and would love to have an alternative leaderboard with harnesses.

sanxiyn 7mo ago
There is. Official leaderboard is without harness, and community leaderboard is with harness. Read ARC-AGI-3 Technical Paper for details.
- falcor84 7mo ago
  I went through the technical paper again, and while they explain why they decided against the harness, I disagree with them - my take is that if harnesses are overfitting, then they should be penalized on the hidden test set. Anyway, searching both in ARC-AGI's paper and website and directly on kaggle, I failed to find a with-harness leaderboard; can you please give the link?
  - sanxiyn 7mo ago
    Here it is: https://arcprize.org/leaderboard/community https://arcprize.org/leaderboard/community
    - falcor84 7mo ago
      Ah, it's based on this repo [0] and there's only 1 non-example submission there [1], from 2 weeks ago (so it only covers the preview games), and their schema doesn't a field to show that it's only the preview, nor does the thing properly parse the score or cost into the table. And the biggest thing is that apparently there's no validation whatsoever - submissions are not ever run on the hidden test games, so is essentially useless as a comparison. [0] https://github.com/arcprize/ARC-AGI-Community-Leaderboard https://github.com/arcprize/ARC-AGI-Community-Leaderboard [1] https://github.com/arcprize/ARC-AGI-Community-Leaderboard/blob/main/submissions/rgb-agent/submission.yaml https://github.com/arcprize/ARC-AGI-Community-Leaderboard/bl...
steve_adams_86 7mo ago
I'm so into harness development right now. Once it clicked that harnesses can bring more safety and determinism to LLMs, I started to wonder where I'd need that and why (vs MCP or just throwing Claude Code at everything), and my brain gears have been turning endlessly since then. I'd love to see more of what people do with them. My use cases are admittedly lame and boring, but it's such a fun paradigm to think and develop around.
- j_bum 7mo ago
  Could you point me to some resources to learn about harnesses? I’d love to hear an example of a use case you’re thinking of.

sanxiyn 7mo ago

There is. Official leaderboard is without harness, and community leaderboard is with harness. Read ARC-AGI-3 Technical Paper for details.

falcor84 7mo ago
I went through the technical paper again, and while they explain why they decided against the harness, I disagree with them - my take is that if harnesses are overfitting, then they should be penalized on the hidden test set. Anyway, searching both in ARC-AGI's paper and website and directly on kaggle, I failed to find a with-harness leaderboard; can you please give the link?
- sanxiyn 7mo ago
  Here it is: https://arcprize.org/leaderboard/community https://arcprize.org/leaderboard/community
  - falcor84 7mo ago
    Ah, it's based on this repo [0] and there's only 1 non-example submission there [1], from 2 weeks ago (so it only covers the preview games), and their schema doesn't a field to show that it's only the preview, nor does the thing properly parse the score or cost into the table. And the biggest thing is that apparently there's no validation whatsoever - submissions are not ever run on the hidden test games, so is essentially useless as a comparison. [0] https://github.com/arcprize/ARC-AGI-Community-Leaderboard https://github.com/arcprize/ARC-AGI-Community-Leaderboard [1] https://github.com/arcprize/ARC-AGI-Community-Leaderboard/blob/main/submissions/rgb-agent/submission.yaml https://github.com/arcprize/ARC-AGI-Community-Leaderboard/bl...

steve_adams_86 7mo ago

I'm so into harness development right now. Once it clicked that harnesses can bring more safety and determinism to LLMs, I started to wonder where I'd need that and why (vs MCP or just throwing Claude Code at everything), and my brain gears have been turning endlessly since then. I'd love to see more of what people do with them. My use cases are admittedly lame and boring, but it's such a fun paradigm to think and develop around.

j_bum 7mo ago
Could you point me to some resources to learn about harnesses? I’d love to hear an example of a use case you’re thinking of.