3 ms·
Whether you suspect OpenAI is gaming benchmarks or not, it is plainly false to assert they were caught gaming benchmarks.
by tedsanders 2y ago
Whether you suspect OpenAI is gaming benchmarks or not, it is plainly false to assert they were caught gaming benchmarks.
- buyucu 2y agothey were caught hiding a financial relationship with a benchmark owner, while making that benchmark a centerpiece of their advertisements. epoch is also very vague about how much access to the dataset openai had, but it's clear that they had more access than the general public: https://www.lesswrong.com/posts/8ZgLYwBmB3vLavjKE/some-lessons-from-the-openai-frontiermath-debacle https://www.lesswrong.com/posts/8ZgLYwBmB3vLavjKE/some-lesso... so yes, openai were caught gaming it.
- tedsanders 2y agoHere's a primary source that denies your allegations: https://x.com/__nmca__/status/1882563755806281986 https://x.com/__nmca__/status/1882563755806281986 OpenAI didn't train on the data or even use it to select a release candidate.
- buyucu 2y agoand why should we believe anything an openai employee says. openai tried to hide their financial relationship with frontier. why do that if you're honest?