3 ms·

I am unable to shake that the Chinese models all perform awfully on the private arc-agi 2 tests.

• osti 7mo ago
  But is arc-agi really that useful though? Nowadays it seems to me that it's just another benchmark that needs to be specifically trained for. Maybe the Chinese models just didn't focus on it as much.
  • sdenton4 7mo ago
    Doing great on public datasets and underperforming on private benchmarks is not a good look.
    • Deegy 7mo ago
      Is it though? Do we still have the expectation that LLMs will eventually be able to solve problems they haven't seen before? Or do we just want the most accurate auto complete at the cheapest price at this point?
      • sdenton4 7mo ago
        It indicates that there's a good chance that they have trained on the test set, making the eval scores useless. Even if you have given up on the dream of generalization entirely, you can't meaningfully compare models which have trained on test to those which have not.
  • stavros 7mo ago
    You're not supposed to train for benchmarks, that's their entire point.