4 ms·
Please don’t spread misinformation. OpenAI didn’t cheat on FrontierMath. They have access to the questions, same as MMLU, MATHD, GPQA, ARC-AGI and pretty much e
by tedsanders 2y ago
Please don’t spread misinformation. OpenAI didn’t cheat on FrontierMath. They have access to the questions, same as MMLU, MATHD, GPQA, ARC-AGI and pretty much every eval. Sure, we could all be lying, but it would be pretty self defeating, horrible for employee morale, and quickly discovered.
- fortyseven 2y agoLiving in a world where consequences no longer matter, this doesn't say much to me. :(
- tedsanders 2y agoWhether you suspect OpenAI is gaming benchmarks or not, it is plainly false to assert they were caught gaming benchmarks.
- buyucu 2y agothey were caught hiding a financial relationship with a benchmark owner, while making that benchmark a centerpiece of their advertisements. epoch is also very vague about how much access to the dataset openai had, but it's clear that they had more access than the general public: https://www.lesswrong.com/posts/8ZgLYwBmB3vLavjKE/some-lessons-from-the-openai-frontiermath-debacle https://www.lesswrong.com/posts/8ZgLYwBmB3vLavjKE/some-lesso... so yes, openai were caught gaming it.
- tedsanders 2y agoHere's a primary source that denies your allegations: https://x.com/__nmca__/status/1882563755806281986 https://x.com/__nmca__/status/1882563755806281986 OpenAI didn't train on the data or even use it to select a release candidate.
- buyucu 2y agoand why should we believe anything an openai employee says. openai tried to hide their financial relationship with frontier. why do that if you're honest?