3 ms·
So many negative comments as if o3 didn’t get 25% on frontiermath - which is absolutely nuts. Sure, LLMs will perform better if the answer to a problem is dire
by whimsicalism 2y ago
So many negative comments as if o3 didn’t get 25% on frontiermath - which is absolutely nuts.
Sure, LLMs will perform better if the answer to a problem is directly in their training set. But that doesn’t mean they perform bad when the answer isn’t in their training set.
- optimalsolver 2y agoEpochAI have to send the questions (but not the answer key) to OpenAI in order to score the models. An overnight 2% -> 25% jump on this benchmark is a bit curious.
- whimsicalism 2y ago1. OpenAI said they did not train on these problems & they don’t train on API calls in general, that is a legal policy. 2. It was a new major model release from work over the course of months - struggle to see that as an ‘overnight’ jump in any real meaning. 3. Why is it easier to believe large scale corporate fraud than that the stated capabilities on a held out test set are real? Reads like cope, if I’m being frank.
- zeroonetwothree 2y agoI don’t think it’s “easier to believe” just that it raises some red flags.
- exitb 2y agoThe 2% result belonged to a traditional LLM that costs cents to run, while o3 is extremely expensive.
- MattDaEskimo 2y agoSure, it did good in frontiermath. That's not what this thread is about. Your comment isn't relevant at all
- whimsicalism 2y agothis thread is about math LLM capability, it’s a bit ridiculous to say that mentioning frontiermath is off topic but that’s just me
- MattDaEskimo 2y agoJust because you can generalize the topic doesn't mean you can ignore the specific conversation and choose your hill to argue. Additionally, the conversation of this topic is about the model's ability to generalize and it's potential overfitting, which is arguably more important than parroting mathematics.
- whimsicalism 2y agoperformance on a held-out set (like frontiermath) compared to putnam (which is not held out) is obviously relevant to a model's potential overfitting. i'm not going to keep replying, others can judge whether they think what i'm saying is "relevant at all."
- MattDaEskimo 2y agoAgain, you set your own goal posts and failed to add any insights. The topic here isn't "o-series sucks", it's addressing a found concern.