3 ms·
OpenAI was caught gaming benchmarks recently with FrontierMath. Just (yet another) sign that benchmarks are very flawed and everyone is training on them. So I
by buyucu 2y ago
OpenAI was caught gaming benchmarks recently with FrontierMath. Just (yet another) sign that benchmarks are very flawed and everyone is training on them.
So I would not put too much weight on how the models are doing on benchmarks.
- brookst 2y agoWas OpenAI demonstrated to have cheated, or just that they had access to the benchmarks and it can’t be proven they didn’t cheat? (Which is hard to do in any case). Last I saw FrontierMath said they had a holdback set of problems specifically to ensure investors with access couldn’t cheat[1]. Or did that turn out to be a lie? 1. https://www.reddit.com/r/singularity/comments/1i4n0r5/this_is_so_disappointing_epoch_ai_the_startup/m7y3x0j/ https://www.reddit.com/r/singularity/comments/1i4n0r5/this_i...
- buyucu 2y agoOpenAI funded the frontier dataset, and hid this relationship. Looks very fishy.
- tedsanders 2y agoPlease don’t spread misinformation. OpenAI didn’t cheat on FrontierMath. They have access to the questions, same as MMLU, MATHD, GPQA, ARC-AGI and pretty much every eval. Sure, we could all be lying, but it would be pretty self defeating, horrible for employee morale, and quickly discovered.
- fortyseven 2y agoLiving in a world where consequences no longer matter, this doesn't say much to me. :(
- tedsanders 2y agoWhether you suspect OpenAI is gaming benchmarks or not, it is plainly false to assert they were caught gaming benchmarks.
- buyucu 2y agothey were caught hiding a financial relationship with a benchmark owner, while making that benchmark a centerpiece of their advertisements. epoch is also very vague about how much access to the dataset openai had, but it's clear that they had more access than the general public: https://www.lesswrong.com/posts/8ZgLYwBmB3vLavjKE/some-lessons-from-the-openai-frontiermath-debacle https://www.lesswrong.com/posts/8ZgLYwBmB3vLavjKE/some-lesso... so yes, openai were caught gaming it.
- tedsanders 2y agoHere's a primary source that denies your allegations: https://x.com/__nmca__/status/1882563755806281986 https://x.com/__nmca__/status/1882563755806281986 OpenAI didn't train on the data or even use it to select a release candidate.
- buyucu 2y agoand why should we believe anything an openai employee says. openai tried to hide their financial relationship with frontier. why do that if you're honest?