4 ms·

The ARC-AGI-3 score is ridiculously high. Is this benchmaxxing or something way different? It's really hard to discern how we're approaching breakthroughs...

• Legend2440 1mo ago
  They explain why here: https://openai.com/index/how-two-settings-tripled-our-arc-agi-3-scores/ https://openai.com/index/how-two-settings-tripled-our-arc-ag... TL;DR all the other models are being crippled by limitations of their harness. >First, we noticed that after each game action, all private reasoning was discarded. This meant that with each action, GPT‑5.6 Sol was asked to figure out the game anew, unable to remember its past thinking. The model could still see a record of past moves and brief accompanying notes, but it could not see the plans, insights, or thoughts that led to them. >Second, we saw that the harness used a rolling truncation window, causing older actions to become invisible as the history grew. So not only was GPT‑5.6 Sol unable to remember its past thinking, it was losing memory of its past actions too.
  • janalsncm 1mo ago
    Ok so the correct comparison would be to fix the harness on the old model and re-compare. Now they are comparing a new model to an old crippled one.
    • altcognito 1mo ago
      They get 66% with the old harness, which is a lot better, but obviously not 100%
    • vlmutolo 1mo ago
      They said 5.6 Sol got something around 40% with the corrected harness.
  • _superposition_ 1mo ago
    Exactly what I suspected. Of course a machine can just iterate relentlessly the way a human can't. I guess token counts are somewhat of a metric. IMO intelligence has peaked and all future gains will come from faster tps and more iteration.
    • deleted 1mo ago
      [deleted]
  • amelius 1mo ago
    Can't AI write harnesses, and why did nobody use AI to figure that out?
• polynomial 1mo ago
  This is absolutely benchmaxxing. Looking forward to hearing from Chollet about it!