5 ms·

Note that this uses a harness so it doesn't qualify for the official ARC-AGI-3 leaderboard According to the authors the harness isn't ARC-AGI specific though h

by lairv 6mo ago

Note that this uses a harness so it doesn't qualify for the official ARC-AGI-3 leaderboard According to the authors the harness isn't ARC-AGI specific though https://x.com/agenticasdk/status/2037335806264971461 https://x.com/agenticasdk/status/2037335806264971461

falcor84 6mo ago
I for one think that harness development is perhaps the most interesting part at the moment and would love to have an alternative leaderboard with harnesses.
- sanxiyn 6mo ago
  There is. Official leaderboard is without harness, and community leaderboard is with harness. Read ARC-AGI-3 Technical Paper for details.
  - falcor84 6mo ago
    I went through the technical paper again, and while they explain why they decided against the harness, I disagree with them - my take is that if harnesses are overfitting, then they should be penalized on the hidden test set. Anyway, searching both in ARC-AGI's paper and website and directly on kaggle, I failed to find a with-harness leaderboard; can you please give the link?
    - sanxiyn 6mo ago
      Here it is: https://arcprize.org/leaderboard/community https://arcprize.org/leaderboard/community
      - falcor84 6mo ago
        Ah, it's based on this repo [0] and there's only 1 non-example submission there [1], from 2 weeks ago (so it only covers the preview games), and their schema doesn't a field to show that it's only the preview, nor does the thing properly parse the score or cost into the table. And the biggest thing is that apparently there's no validation whatsoever - submissions are not ever run on the hidden test games, so is essentially useless as a comparison. [0] https://github.com/arcprize/ARC-AGI-Community-Leaderboard https://github.com/arcprize/ARC-AGI-Community-Leaderboard [1] https://github.com/arcprize/ARC-AGI-Community-Leaderboard/blob/main/submissions/rgb-agent/submission.yaml https://github.com/arcprize/ARC-AGI-Community-Leaderboard/bl...
- steve_adams_86 6mo ago
  I'm so into harness development right now. Once it clicked that harnesses can bring more safety and determinism to LLMs, I started to wonder where I'd need that and why (vs MCP or just throwing Claude Code at everything), and my brain gears have been turning endlessly since then. I'd love to see more of what people do with them. My use cases are admittedly lame and boring, but it's such a fun paradigm to think and develop around.
  - j_bum 6mo ago
    Could you point me to some resources to learn about harnesses? I’d love to hear an example of a use case you’re thinking of.
krackers 6mo ago
> this uses a harness This seems like an arbitrary restriction. Tool-use requires a harness, and their whitepaper never defines exactly what counts as valid.
- fermentation 6mo ago
  Right, fair, but look at the prompt. For the purpose of testing general intelligence, this seems kind of pointless.
- UltraSane 6mo ago
  It isn't arbitrary. They want measure the capability of the general LLM
  - fc417fc802 6mo ago
    So if I say "I want to measure your capability as a mechanic" but then also "to ensure an accurate score you're forbidden to use any tools" how are you the human mechanic planning to diagnose and fix the engine problem without wrenches and jack stands and the like? It makes no sense. That said their harness isn't generic. It includes a ridiculously detailed prompt for how to play this specific game. Forbidding tool use is arbitrary and above all pointless hoop jumping but that doesn't make the linked "achievement" any less fraudulent.
    - UltraSane 6mo ago
      It is more like restricting the mechanic to only using commercially available tools and not allow them to create CUSTOM tools.
      - fc417fc802 6mo ago
        No, that would be analogous to disallowing customized harnesses, ie tooling specially crafted by someone else for the specific task at hand. Insisting that an LLM solve something without the ability to make use of any external tooling whatsoever is almost perfectly analogous to insisting that a human mechanic work on a car with nothing but his own bare hands. The wrench is to the mechanic as the stock python repl is to the LLM.
osti 6mo ago
Doesn't the chat version of chatgpt or gemini also have interleaved tool calls, so do those also count as with harnesses?
- WiSaGaN 6mo ago
  Harness is fine. I think people here are arguing what provided here to take the test is not harness.
fchollet 6mo ago
It is 100% ARC-AGI-3 specific though, just read through the prompts https://github.com/symbolica-ai/ARC-AGI-3-Agents/blob/symbolica/arcgentica/agents/templates/agentica/prompts.py https://github.com/symbolica-ai/ARC-AGI-3-Agents/blob/symbol...
- DetroitThrow 6mo ago
  Um, yes this is a extremely specific as a benchmark harness. It has a ton of knowledge encoded about the tasks at hand. The tweet is dishonest even in the best light. The hard part of these tests isn't purely reasoning ability ffs.
- boxed 6mo ago
  What a dick move. Making that prompt open source will probably mean that every other model that doesn't want to cheat will scrape that and accidentally cheat in the next models.
- diwank 6mo ago
  this is so disingenuous on symbolica's part. these insincere announcements just make it harder for genuine attempts and novel ideas
- cxdorn 6mo ago
  (disclaimer: i worked on early versions of agentica_sdk; but wasn't involved in recent developments and the ARC solver) As other comments point out this is about harness development and harness efficiency. Agentica SDK is a sort of meta harness, that makes things easy: plug any "internal API" (as defined natively in your codebase) directly into your agent. Agentica SDK itself is not application specifc; but the APIs of your application are... application specific. Re: the linked prompt. A harness is a set of tools and descriptions how to best use those tools, and sometimes some external control flow based on the outcome of using those tools. How to "best use the tools" should always be part of the prompt (like in this case). So this work tries to answer: "short of telling the agent any solutions, make a simple but efficient API to play the games, hand it to the agent, and see how it does". In the world of harness development I think that's an interesting question to answer!
mmaunder 6mo ago
We're calling agents harnesses now?
- fritzo 6mo ago
  ELI5 what is a harness? EDIT from https://arcprize.org/media/ARC_AGI_3_Technical_Report.pdf https://arcprize.org/media/ARC_AGI_3_Technical_Report.pdf: > We seek to fight two forms of overfitting that would muddy public sensefinding: > Task-specific overfitting. This includes any agent that is created with knowledge of public ARC-AGI-3 environments, subsequently being evaluated on the same environments. It could be either directly trained on these environments, or using a harness that is handcrafted or specifically configured by someone with knowledge of the public environments.
- lwansbrough 6mo ago
  I think generally people regard a harness as the system instructions + tools made available to the LLM (and probably the thing that runs the LLM conversation in a loop.) An agent is collectively, the LLM plus the harness.
- boxed 6mo ago
  The point of this test is to check if an AI system can figure out the game. This isn't what happened here. A human figured out the game, wrote in their prompts exactly how the game works and THEN put the AI on the problem. This is 100% cheating and imo quite stupid.
  - ithkuil 6mo ago
    The harness would be fine if the agent coded its own harness in a controlled environment while observing the game. Not sure if the specific rules of this prize allow that, but I would accept that

by lairv 6mo ago

falcor84 6mo ago

I for one think that harness development is perhaps the most interesting part at the moment and would love to have an alternative leaderboard with harnesses.

sanxiyn 6mo ago
There is. Official leaderboard is without harness, and community leaderboard is with harness. Read ARC-AGI-3 Technical Paper for details.
- falcor84 6mo ago
  I went through the technical paper again, and while they explain why they decided against the harness, I disagree with them - my take is that if harnesses are overfitting, then they should be penalized on the hidden test set. Anyway, searching both in ARC-AGI's paper and website and directly on kaggle, I failed to find a with-harness leaderboard; can you please give the link?
  - sanxiyn 6mo ago
    Here it is: https://arcprize.org/leaderboard/community https://arcprize.org/leaderboard/community
    - falcor84 6mo ago
      Ah, it's based on this repo [0] and there's only 1 non-example submission there [1], from 2 weeks ago (so it only covers the preview games), and their schema doesn't a field to show that it's only the preview, nor does the thing properly parse the score or cost into the table. And the biggest thing is that apparently there's no validation whatsoever - submissions are not ever run on the hidden test games, so is essentially useless as a comparison. [0] https://github.com/arcprize/ARC-AGI-Community-Leaderboard https://github.com/arcprize/ARC-AGI-Community-Leaderboard [1] https://github.com/arcprize/ARC-AGI-Community-Leaderboard/blob/main/submissions/rgb-agent/submission.yaml https://github.com/arcprize/ARC-AGI-Community-Leaderboard/bl...
steve_adams_86 6mo ago
I'm so into harness development right now. Once it clicked that harnesses can bring more safety and determinism to LLMs, I started to wonder where I'd need that and why (vs MCP or just throwing Claude Code at everything), and my brain gears have been turning endlessly since then. I'd love to see more of what people do with them. My use cases are admittedly lame and boring, but it's such a fun paradigm to think and develop around.
- j_bum 6mo ago
  Could you point me to some resources to learn about harnesses? I’d love to hear an example of a use case you’re thinking of.

krackers 6mo ago

> this uses a harness This seems like an arbitrary restriction. Tool-use requires a harness, and their whitepaper never defines exactly what counts as valid.

fermentation 6mo ago
Right, fair, but look at the prompt. For the purpose of testing general intelligence, this seems kind of pointless.
UltraSane 6mo ago
It isn't arbitrary. They want measure the capability of the general LLM
- fc417fc802 6mo ago
  So if I say "I want to measure your capability as a mechanic" but then also "to ensure an accurate score you're forbidden to use any tools" how are you the human mechanic planning to diagnose and fix the engine problem without wrenches and jack stands and the like? It makes no sense. That said their harness isn't generic. It includes a ridiculously detailed prompt for how to play this specific game. Forbidding tool use is arbitrary and above all pointless hoop jumping but that doesn't make the linked "achievement" any less fraudulent.
  - UltraSane 6mo ago
    It is more like restricting the mechanic to only using commercially available tools and not allow them to create CUSTOM tools.
    - fc417fc802 6mo ago
      No, that would be analogous to disallowing customized harnesses, ie tooling specially crafted by someone else for the specific task at hand. Insisting that an LLM solve something without the ability to make use of any external tooling whatsoever is almost perfectly analogous to insisting that a human mechanic work on a car with nothing but his own bare hands. The wrench is to the mechanic as the stock python repl is to the LLM.

osti 6mo ago

Doesn't the chat version of chatgpt or gemini also have interleaved tool calls, so do those also count as with harnesses?

WiSaGaN 6mo ago
Harness is fine. I think people here are arguing what provided here to take the test is not harness.

fchollet 6mo ago

It is 100% ARC-AGI-3 specific though, just read through the prompts https://github.com/symbolica-ai/ARC-AGI-3-Agents/blob/symbolica/arcgentica/agents/templates/agentica/prompts.py https://github.com/symbolica-ai/ARC-AGI-3-Agents/blob/symbol...

DetroitThrow 6mo ago
Um, yes this is a extremely specific as a benchmark harness. It has a ton of knowledge encoded about the tasks at hand. The tweet is dishonest even in the best light. The hard part of these tests isn't purely reasoning ability ffs.
boxed 6mo ago
What a dick move. Making that prompt open source will probably mean that every other model that doesn't want to cheat will scrape that and accidentally cheat in the next models.
diwank 6mo ago
this is so disingenuous on symbolica's part. these insincere announcements just make it harder for genuine attempts and novel ideas
cxdorn 6mo ago
(disclaimer: i worked on early versions of agentica_sdk; but wasn't involved in recent developments and the ARC solver) As other comments point out this is about harness development and harness efficiency. Agentica SDK is a sort of meta harness, that makes things easy: plug any "internal API" (as defined natively in your codebase) directly into your agent. Agentica SDK itself is not application specifc; but the APIs of your application are... application specific. Re: the linked prompt. A harness is a set of tools and descriptions how to best use those tools, and sometimes some external control flow based on the outcome of using those tools. How to "best use the tools" should always be part of the prompt (like in this case). So this work tries to answer: "short of telling the agent any solutions, make a simple but efficient API to play the games, hand it to the agent, and see how it does". In the world of harness development I think that's an interesting question to answer!

mmaunder 6mo ago

We're calling agents harnesses now?

fritzo 6mo ago
ELI5 what is a harness? EDIT from https://arcprize.org/media/ARC_AGI_3_Technical_Report.pdf https://arcprize.org/media/ARC_AGI_3_Technical_Report.pdf: > We seek to fight two forms of overfitting that would muddy public sensefinding: > Task-specific overfitting. This includes any agent that is created with knowledge of public ARC-AGI-3 environments, subsequently being evaluated on the same environments. It could be either directly trained on these environments, or using a harness that is handcrafted or specifically configured by someone with knowledge of the public environments.
lwansbrough 6mo ago
I think generally people regard a harness as the system instructions + tools made available to the LLM (and probably the thing that runs the LLM conversation in a loop.) An agent is collectively, the LLM plus the harness.
boxed 6mo ago
The point of this test is to check if an AI system can figure out the game. This isn't what happened here. A human figured out the game, wrote in their prompts exactly how the game works and THEN put the AI on the problem. This is 100% cheating and imo quite stupid.
- ithkuil 6mo ago
  The harness would be fine if the agent coded its own harness in a controlled environment while observing the game. Not sure if the specific rules of this prize allow that, but I would accept that