3 ms·

The fact that this was on the set of training problems with a custom harness basically makes the headline a lie. What if you give opus the same harness? Do peo

The fact that this was on the set of training problems with a custom harness basically makes the headline a lie. What if you give opus the same harness? Do people even care about meaningful comparisons any more or is it all just “numbers go up”

cbg0 7mo ago
When you're on the hunt for VC cash "numbers go up" is the main criteria.
ting0 7mo ago
Does it matter though? If it accomplishes the task, it accomplishes the task. Everyone uses a harness anyway, and finding the best harness is relevant. Also perhaps this hints at something bigger, i.e.: we're wasting our time focusing on the model when we could be focusing on the harness.
- stephantul 7mo ago
  Yes it matters, because it’s not a measurement of whether it accomplishes the task if a human tells it how to solve it.
fzeindl 7mo ago
Would the single sentence „Imagine you are a regular computer player and accustomed to the usual elements of games“ count as a harness?
versteegen 7mo ago
...Their agent is called "Agentica ARC-AGI-3 agent for Opus 4.6 (120k) High". Yes, it's unfair to compare results for the 25 (easier) public games against scores for the 55 semi-private games (scores for which are taken from https://arcprize.org/leaderboard https://arcprize.org/leaderboard). But you're wrong to say that a custom harness invalidates the result. Yes, the official "ARC verified" scoreboard for frontier LLMs requires (https://arcprize.org/policy https://arcprize.org/policy): > using extremely generic and miminal LLM testing prompts, no client-side "harnesses", no hand-crafted tools, and no tailored model configuration but these are limitations placed in order to compare LLMs from frontier labs on equal footing, not limitations that apply to submissions in general. It's not as if a solution to ARC-AGI-3 must involve training a custom LLM! This Agentica harness is completely legitimate approach to ARC-AGI-3, similar to J. Berman's for ARC-AGI-1/2, for example.
- stephantul 7mo ago
  I’m not saying it invalidates the result. I am saying that they knew the headline and comparison was not correct and they still decided to roll with it. It’s an incorrect representation of what happened, designed to get eyeballs and possibly vc dollars.

cbg0 7mo ago

When you're on the hunt for VC cash "numbers go up" is the main criteria.

ting0 7mo ago

Does it matter though? If it accomplishes the task, it accomplishes the task. Everyone uses a harness anyway, and finding the best harness is relevant. Also perhaps this hints at something bigger, i.e.: we're wasting our time focusing on the model when we could be focusing on the harness.

stephantul 7mo ago
Yes it matters, because it’s not a measurement of whether it accomplishes the task if a human tells it how to solve it.

fzeindl 7mo ago

Would the single sentence „Imagine you are a regular computer player and accustomed to the usual elements of games“ count as a harness?

versteegen 7mo ago

...Their agent is called "Agentica ARC-AGI-3 agent for Opus 4.6 (120k) High". Yes, it's unfair to compare results for the 25 (easier) public games against scores for the 55 semi-private games (scores for which are taken from https://arcprize.org/leaderboard https://arcprize.org/leaderboard). But you're wrong to say that a custom harness invalidates the result. Yes, the official "ARC verified" scoreboard for frontier LLMs requires (https://arcprize.org/policy https://arcprize.org/policy): > using extremely generic and miminal LLM testing prompts, no client-side "harnesses", no hand-crafted tools, and no tailored model configuration but these are limitations placed in order to compare LLMs from frontier labs on equal footing, not limitations that apply to submissions in general. It's not as if a solution to ARC-AGI-3 must involve training a custom LLM! This Agentica harness is completely legitimate approach to ARC-AGI-3, similar to J. Berman's for ARC-AGI-1/2, for example.

stephantul 7mo ago
I’m not saying it invalidates the result. I am saying that they knew the headline and comparison was not correct and they still decided to roll with it. It’s an incorrect representation of what happened, designed to get eyeballs and possibly vc dollars.