11 ms·
Day 1 of ARC-AGI-3
- lairv 6mo agoNote that this uses a harness so it doesn't qualify for the official ARC-AGI-3 leaderboard According to the authors the harness isn't ARC-AGI specific though https://x.com/agenticasdk/status/2037335806264971461 https://x.com/agenticasdk/status/2037335806264971461
- falcor84 6mo agoI for one think that harness development is perhaps the most interesting part at the moment and would love to have an alternative leaderboard with harnesses.
- sanxiyn 6mo agoThere is. Official leaderboard is without harness, and community leaderboard is with harness. Read ARC-AGI-3 Technical Paper for details.
- falcor84 6mo agoI went through the technical paper again, and while they explain why they decided against the harness, I disagree with them - my take is that if harnesses are overfitting, then they should be penalized on the hidden test set. Anyway, searching both in ARC-AGI's paper and website and directly on kaggle, I failed to find a with-harness leaderboard; can you please give the link?
- sanxiyn 6mo agoHere it is: https://arcprize.org/leaderboard/community https://arcprize.org/leaderboard/community
- falcor84 6mo agoAh, it's based on this repo [0] and there's only 1 non-example submission there [1], from 2 weeks ago (so it only covers the preview games), and their schema doesn't a field to show that it's only the preview, nor does the thing properly parse the score or cost into the table. And the biggest thing is that apparently there's no validation whatsoever - submissions are not ever run on the hidden test games, so is essentially useless as a comparison. [0] https://github.com/arcprize/ARC-AGI-Community-Leaderboard https://github.com/arcprize/ARC-AGI-Community-Leaderboard [1] https://github.com/arcprize/ARC-AGI-Community-Leaderboard/blob/main/submissions/rgb-agent/submission.yaml https://github.com/arcprize/ARC-AGI-Community-Leaderboard/bl...
- steve_adams_86 6mo agoI'm so into harness development right now. Once it clicked that harnesses can bring more safety and determinism to LLMs, I started to wonder where I'd need that and why (vs MCP or just throwing Claude Code at everything), and my brain gears have been turning endlessly since then. I'd love to see more of what people do with them. My use cases are admittedly lame and boring, but it's such a fun paradigm to think and develop around.
- j_bum 6mo agoCould you point me to some resources to learn about harnesses? I’d love to hear an example of a use case you’re thinking of.
- krackers 6mo ago> this uses a harness This seems like an arbitrary restriction. Tool-use requires a harness, and their whitepaper never defines exactly what counts as valid.
- fermentation 6mo agoRight, fair, but look at the prompt. For the purpose of testing general intelligence, this seems kind of pointless.
- UltraSane 6mo agoIt isn't arbitrary. They want measure the capability of the general LLM
- fc417fc802 6mo agoSo if I say "I want to measure your capability as a mechanic" but then also "to ensure an accurate score you're forbidden to use any tools" how are you the human mechanic planning to diagnose and fix the engine problem without wrenches and jack stands and the like? It makes no sense. That said their harness isn't generic. It includes a ridiculously detailed prompt for how to play this specific game. Forbidding tool use is arbitrary and above all pointless hoop jumping but that doesn't make the linked "achievement" any less fraudulent.
- UltraSane 6mo agoIt is more like restricting the mechanic to only using commercially available tools and not allow them to create CUSTOM tools.
- fc417fc802 6mo agoNo, that would be analogous to disallowing customized harnesses, ie tooling specially crafted by someone else for the specific task at hand. Insisting that an LLM solve something without the ability to make use of any external tooling whatsoever is almost perfectly analogous to insisting that a human mechanic work on a car with nothing but his own bare hands. The wrench is to the mechanic as the stock python repl is to the LLM.
- osti 6mo agoDoesn't the chat version of chatgpt or gemini also have interleaved tool calls, so do those also count as with harnesses?
- WiSaGaN 6mo agoHarness is fine. I think people here are arguing what provided here to take the test is not harness.
- fchollet 6mo agoIt is 100% ARC-AGI-3 specific though, just read through the prompts https://github.com/symbolica-ai/ARC-AGI-3-Agents/blob/symbolica/arcgentica/agents/templates/agentica/prompts.py https://github.com/symbolica-ai/ARC-AGI-3-Agents/blob/symbol...
- DetroitThrow 6mo agoUm, yes this is a extremely specific as a benchmark harness. It has a ton of knowledge encoded about the tasks at hand. The tweet is dishonest even in the best light. The hard part of these tests isn't purely reasoning ability ffs.
- boxed 6mo agoWhat a dick move. Making that prompt open source will probably mean that every other model that doesn't want to cheat will scrape that and accidentally cheat in the next models.
- diwank 6mo agothis is so disingenuous on symbolica's part. these insincere announcements just make it harder for genuine attempts and novel ideas
- cxdorn 6mo ago(disclaimer: i worked on early versions of agentica_sdk; but wasn't involved in recent developments and the ARC solver) As other comments point out this is about harness development and harness efficiency. Agentica SDK is a sort of meta harness, that makes things easy: plug any "internal API" (as defined natively in your codebase) directly into your agent. Agentica SDK itself is not application specifc; but the APIs of your application are... application specific. Re: the linked prompt. A harness is a set of tools and descriptions how to best use those tools, and sometimes some external control flow based on the outcome of using those tools. How to "best use the tools" should always be part of the prompt (like in this case). So this work tries to answer: "short of telling the agent any solutions, make a simple but efficient API to play the games, hand it to the agent, and see how it does". In the world of harness development I think that's an interesting question to answer!
- mmaunder 6mo agoWe're calling agents harnesses now?
- fritzo 6mo agoELI5 what is a harness? EDIT from https://arcprize.org/media/ARC_AGI_3_Technical_Report.pdf https://arcprize.org/media/ARC_AGI_3_Technical_Report.pdf: > We seek to fight two forms of overfitting that would muddy public sensefinding: > Task-specific overfitting. This includes any agent that is created with knowledge of public ARC-AGI-3 environments, subsequently being evaluated on the same environments. It could be either directly trained on these environments, or using a harness that is handcrafted or specifically configured by someone with knowledge of the public environments.
- lwansbrough 6mo agoI think generally people regard a harness as the system instructions + tools made available to the LLM (and probably the thing that runs the LLM conversation in a loop.) An agent is collectively, the LLM plus the harness.
- boxed 6mo agoThe point of this test is to check if an AI system can figure out the game. This isn't what happened here. A human figured out the game, wrote in their prompts exactly how the game works and THEN put the AI on the problem. This is 100% cheating and imo quite stupid.
- ithkuil 6mo agoThe harness would be fine if the agent coded its own harness in a controlled environment while observing the game. Not sure if the specific rules of this prize allow that, but I would accept that
- esafak 6mo agoAnybody used this Agentica of theirs?
- AbanoubRodolf 6mo ago[flagged]
- modeless 6mo agoOn the public set of 25 problems. These are intended for development and testing, not evaluation. There are 110 private problems for actual evaluation purposes, and the ARC-AGI-3 paper says "the public set is materially easier than the private set".
- SchemaLoad 6mo agoBenchmarks on public tests are too easy to game. The model owners can just incorporate the answers in to the dataset. Only the private problems actually matter.
- sanxiyn 6mo agoIn this case the code is public and you can see they are not cheating in that sense.
- SchemaLoad 6mo agoOnce the model has seen the questions and answers in the training stage, the questions are worthless. Only a test using previously unseen questions has merit.
- lambda 6mo agoThey aren't training new models for this. This is an agent harness for Opus 4.6.
- measurablefunc 6mo agoAll traffic is monitored, all signal sources are eventually incorporated into the training set in one way or another. The person you're responding to is correct, even a single API call to any AI provider is sufficient to discount future results from the same provider.
- 6mo ago
- gslin 6mo agohttps://en.wikipedia.org/wiki/Goodhart's_law https://en.wikipedia.org/wiki/Goodhart's_law > Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.
- deleted 6mo ago[deleted]
- bytesandbits 6mo agowe constantly underestimate the power of inference scaffolding. I have seen it in all domains: coding, ASR, ARC-AGI benchmarks you name it. Scaffolding can do a lot! And post-training too. I am confident our currently pre-trained models can beat this benchmark over 80% with the right post-training and scaffolding. That being said I don't think ARC-AGI proves much. It is not a useful task at all in the wild. it is just a game; a strange and confusing one. For me this is just a pointless pseudo-academic exercise. Good to have, but by no means measures intelligence and even less utility of a model.
- nubg 6mo agowhat exactly does scaffolding mean in this context? genuine question
- bytesandbits 6mo agoanything that doesn't touch the model parameters at all once it has been compiled. for example, in streaming ASR of an encoder-decoder you can get gains in accuracy just by enhancing the encoder-decoder orchestration and ratio, frequency of fwd passes, dynamically adjusting the length of rolling windows (if using full attention). Prompting would be part of this too, including few-shot examples. Decoding strategy is also part of this (top-k, nucleus, speculative decoding, greedy or anything else). Applying signal processing or any kind of processing to the input before getting it into the model, or to the output. There are a lot of things you can do.
- Linello 6mo agoAlso think about the program-synthesis approach proposed by Poetiq.ai. python programs are being generated and evaluated against previous examples. Then in-context learning is done programmatically via prompt concatenation. If you can "score" online the working and non working examples, then you have a very strong reward signal.
- boxed 6mo agoI'm gonna guess it means "whatever we still need humans to figure out to spoon feed the models"
- stephantul 6mo agoThe fact that this was on the set of training problems with a custom harness basically makes the headline a lie. What if you give opus the same harness? Do people even care about meaningful comparisons any more or is it all just “numbers go up”
- cbg0 6mo agoWhen you're on the hunt for VC cash "numbers go up" is the main criteria.
- ting0 6mo agoDoes it matter though? If it accomplishes the task, it accomplishes the task. Everyone uses a harness anyway, and finding the best harness is relevant. Also perhaps this hints at something bigger, i.e.: we're wasting our time focusing on the model when we could be focusing on the harness.
- stephantul 6mo agoYes it matters, because it’s not a measurement of whether it accomplishes the task if a human tells it how to solve it.
- fzeindl 6mo agoWould the single sentence „Imagine you are a regular computer player and accustomed to the usual elements of games“ count as a harness?
- versteegen 6mo ago...Their agent is called "Agentica ARC-AGI-3 agent for Opus 4.6 (120k) High". Yes, it's unfair to compare results for the 25 (easier) public games against scores for the 55 semi-private games (scores for which are taken from https://arcprize.org/leaderboard https://arcprize.org/leaderboard). But you're wrong to say that a custom harness invalidates the result. Yes, the official "ARC verified" scoreboard for frontier LLMs requires (https://arcprize.org/policy https://arcprize.org/policy): > using extremely generic and miminal LLM testing prompts, no client-side "harnesses", no hand-crafted tools, and no tailored model configuration but these are limitations placed in order to compare LLMs from frontier labs on equal footing, not limitations that apply to submissions in general. It's not as if a solution to ARC-AGI-3 must involve training a custom LLM! This Agentica harness is completely legitimate approach to ARC-AGI-3, similar to J. Berman's for ARC-AGI-1/2, for example.
- padolsey 6mo agoKnowing the nature of a test ahead of time, building out your capabilities and tooling before entering the exam hall when your peers don't have that advantage, makes you a cheater.
- BoorishBears 6mo agoLots of people doing the same with extra steps (generating synthetic data from test questions with the LLM then training on it) I wish we'd move past public test sets for LLM benchmarks: publish a plain english explanation of the tasks, allow questions and clarifications, and but never release a single question from the test set verbatim. It made sense back when models needed to be finetuned on the task to even reliably answer. If we're saying this is the path to AGI we should be able to rely on the generalization of the model to get it right.
- ting0 6mo agoYou have a problem with generating synthetic data from test questions? Humans simulate experiences in their mind. What's the problem?
- BoorishBears 6mo agoModels don't generalize as well as humans. If a model was trained on <|begin_text|> <|end_text|> and you change the tokens passed to <|start_text|> <|end_text|>, it loses several 'IQ points' if it can even answer back at all anymore. Synthetic data is fine. Synthetic data on very similar questions generated based on the description is typically fine. But once the shape of what you're training on gets too close to the actual holdout questions, you're getting an uplift that's not realistic for unseen tasks.
- GorbachevyChase 6mo agoHumans who have played games should also not be allowed to test in ARC AGI. Cavemen only.
- mohsen1 6mo agoUses public dataset to evaluate which is not meant for evaluation. Writes super specific prompt[1] and claims eye catching results. This is the state of "AI" these days I guess... [1] https://github.com/symbolica-ai/ARC-AGI-3-Agents/blob/symbolica/arcgentica/agents/templates/agentica/prompts.py https://github.com/symbolica-ai/ARC-AGI-3-Agents/blob/symbol...
- Rebuff5007 6mo agoOf course it is... we are in an era where a well-timed blog post showing "SOTA results" on a benchmark can net millions in funding
- versteegen 6mo agoThe dataset miscomparison is a big problem. The prompt is super specific to ARC-AGI-3, which is perfectly fine to do, but skimming it I saw nothing that appears specific to the 25 games in the dataset. Especially considering they've only had one day for overfitting. Could be quite subtle leakage though.
- andy12_ 6mo agoApparently the score would be a little higher if it weren't for the fact that scores are penalized for being worse than the human baseline, but aren't rewarded for being better than the human baseline (which seems like an arbitrary decision. The human baseline is not optimal).
- modeless 6mo agoOnce you have matched humans on a problem then further progress on that problem is not necessarily meaningful anymore, in terms of quantitative measurement of intelligence. ARC-AGI-3 is designed to compare AIs to humans, not to measure arbitrarily high levels of superhuman intelligence. For that you would want a different benchmark.
- dsfadfasdf 6mo agoCan someone clarify if image inputs are allowed, so VLMs can be used? I have not been able to get information anywhere.