3 ms·
EpochAI have to send the questions (but not the answer key) to OpenAI in order to score the models. An overnight 2% -> 25% jump on this benchmark is a bit curi
by optimalsolver 2y ago
EpochAI have to send the questions (but not the answer key) to OpenAI in order to score the models.
An overnight 2% -> 25% jump on this benchmark is a bit curious.
- whimsicalism 2y ago1. OpenAI said they did not train on these problems & they don’t train on API calls in general, that is a legal policy. 2. It was a new major model release from work over the course of months - struggle to see that as an ‘overnight’ jump in any real meaning. 3. Why is it easier to believe large scale corporate fraud than that the stated capabilities on a held out test set are real? Reads like cope, if I’m being frank.
- zeroonetwothree 2y agoI don’t think it’s “easier to believe” just that it raises some red flags.
- exitb 2y agoThe 2% result belonged to a traditional LLM that costs cents to run, while o3 is extremely expensive.