Anytime you see that, we should assume the newer models might have been trained on either the benchmarks themselves or something similar to them. If I was an ev

Anytime you see that, we should assume the newer models might have been trained on either the benchmarks themselves or something similar to them. If I was an evaluator, I’d keep a secret pile of tests that I know aren’t in any LLM’s, do the evaluations privately, and not publish scores either. Just rank plus how far apart they are. The best tests of these models are people who want to use AI to solve real problems attempting to do that with various models. If they work, report that they worked. Also, publish the work and result pairs permissively when possible to evaluate that and use it for fine-tuning, too.