Thanks for the interesting question! From what I've observed from both academia and posts from a number of top organizations (with seemingly the exception of O

Thanks for the interesting question! From what I've observed from both academia and posts from a number of top organizations (with seemingly the exception of OpenAI) - all the models are trained on test data. That's why OpenAI seems to report poorer results than the rest but perform the best in practice - they seem to have their own curated dataset. I have arguably the opposite opinion here - I don't think we can "prevent" cheating but we should instead put in the work to create new exam questions and make it easier to do so. I feel "learning" the exam questions is (IMO) completely fine and should be expected (why should we have a static dataset if it diminishes the quality of the results) - much like having new exam papers every year for students. What I expect going forward is for the next version of foundation models, there will also be smarter ways to synthetically create datasets from ground truths themselves (we made our first pass at it here - https://docs.confident-ai.com/docs/quickstart/synthetic-data-creation https://docs.confident-ai.com/docs/quickstart/synthetic-data... ). I feel that data labelling companies will hold an edge and that the cycle of finding new edge cases, labelling, continue training will continue in the foreseeable future (see Karpathy's data engine flywheel in 2021/2022 Tesla AI day).