I am suggesting that it is only logical for a company whose main advertising comes from good benchmark numbers to play games with the benchmarks. In this case,

I am suggesting that it is only logical for a company whose main advertising comes from good benchmark numbers to play games with the benchmarks. In this case, I am suggesting that they run a fine-tuning/RL pass using benchmark scores as an objective function or using a training set that otherwise looks a lot like the benchmarks. Every single other company whose marketing depends on benchmarks does the analogue of this to some degree. And we won't know for sure that they aren't doing this until they publicly disclose details about their model and training process (like every other research org does), allowing other researchers to run replication studies. Also, I don't appreciate the ad hominems. Comments about some unrelated "conspiracy theorist" and "vaccine discourse" add nothing to the discussion.