3 ms·

We're running out of benchmarks to upper bound AI capabilities

• WarmWash 7mo ago
  Start front loading the models with 5k, 10k, 50k, 100k tokens of messy quasi related context, and then run the benchmarks. These models are ridiculously powerful with a blank slate. It's when they get loaded down with all the necessary (and inevitably unnecessary) context to complete the task that they really start to crumble and fold.
  • jballanc 7mo ago
    We need benchmarks that can distinguish between continuous learning and long-context extrapolation.
    • vrighter 7mo ago
      oh that's easy: continuous learning is not something current architectures can do. So the benchmark for that can be done mentally
• nikisweeting 7mo ago
  We can definitely make harder evals, the problem is a good eval set is indistinguishable from good training data / market edge, so no one is incentivized to share their best eval sets publicly.
• UltraSane 7mo ago
  This is the least true thing ever. All LLMs are terrible at ARC-AGI-3. Every video game can be used as a benchmark. You could rank LLMs on how long they can keep a game of Dwarf Fortress running or how fast they can beat GTA5.
  • ttoinou 7mo ago
    We already have specialized AI to play video games
    • UltraSane 7mo ago
      We are talking about LLMs. a true AGI would be able to beat every video game.
      • conception 7mo ago
        Until Arc-Battletoads is passed I’m not buying it.
        • UltraSane 7mo ago
          More like ARC-SegaMasterSystem-ALF
  • casey2 7mo ago
    ARC-AGI-3 appears to already be saturated [0] [1] For some reason they refuse to run this on the private set, likely because it's all just a ploy to pump OpenAI [0] https://arcprize.org/leaderboard/community https://arcprize.org/leaderboard/community [1] https://blog.alexisfox.dev/arcagi3 https://blog.alexisfox.dev/arcagi3
• refactorbench 7mo ago
  [dead]