3 ms·

This study appears to be evidence against that: the model failed the benchmark but arrived at the correct answer.

• fsh 19d ago
  Half of the benchmarks don't have published answers. That's why training data providers have been hiring physicists to solve them.
  • deleted 19d ago
    [deleted]
  • red75prime 19d ago
    This is unconventional benchmaxxing then, when they decrease the benchmark scores to allow models to generalize on correct solutions.