4 ms·

Living in a world where consequences no longer matter, this doesn't say much to me. :(

• tedsanders 2y ago
  Whether you suspect OpenAI is gaming benchmarks or not, it is plainly false to assert they were caught gaming benchmarks.
  • buyucu 2y ago
    they were caught hiding a financial relationship with a benchmark owner, while making that benchmark a centerpiece of their advertisements. epoch is also very vague about how much access to the dataset openai had, but it's clear that they had more access than the general public: https://www.lesswrong.com/posts/8ZgLYwBmB3vLavjKE/some-lessons-from-the-openai-frontiermath-debacle https://www.lesswrong.com/posts/8ZgLYwBmB3vLavjKE/some-lesso... so yes, openai were caught gaming it.
    • tedsanders 2y ago
      Here's a primary source that denies your allegations: https://x.com/__nmca__/status/1882563755806281986 https://x.com/__nmca__/status/1882563755806281986 OpenAI didn't train on the data or even use it to select a release candidate.
      • buyucu 2y ago
        and why should we believe anything an openai employee says. openai tried to hide their financial relationship with frontier. why do that if you're honest?