3 ms·
o3 is able to get 25% on never seen before frontiermath problems. sure, the models do better when the answer is directly in their dataset but they’ve already su
by whimsicalism 2y ago
o3 is able to get 25% on never seen before frontiermath problems. sure, the models do better when the answer is directly in their dataset but they’ve already surpassed the average human in novelty on held out problems
- jvanderbot 2y agoThe average human did zero studying on representative problems. LLMs did a lot.
- whimsicalism 2y agoOkay? We are measuring capabilities.
- tzs 2y agoI don't know anything about frontiermath problems, but for Putnam problems (which is what the submitted article is about) the average human that takes the exam is an undergraduate mathematics or science major who has studied prior Putnam problems and other similar problems recently to specifically prepare for the exam...and the most common score is still 0. At top tier schools the most common score will usually be somewhere in the 0 to 10 range (out of a possible 120).
- gldjmp 2y agoYou can get all of the correct numerical answers for the Putnam and still get a zero, because the reasoning is graded very harshly. The scores measured in this paper are not comparable to actual Putnam scores.
- fldskfjdslkfj 2y ago> never seen before frontiermath problems How do you know that?
- whimsicalism 2y agoBecause that is the whole conceit of how frontiermath is constructed
- fldskfjdslkfj 2y agoDidn't they run a bunch of models on the problem set? I doubt they are hosting all those models on their own infrastructure.
- whimsicalism 2y ago1. OpenAI has confirmed it’s not in their train (unlike putnam where they have never made any such claims) 2. They don't train on API calls 3. It is funny to me that HN finds it easier to believe theories about stealing data from APIs rather than an improvement in capabilities. It would be nice if symmetric scrutiny were applied to optimistic and pessimistic claims about LLMs, but I certainly don’t feel that is the case here.
- fldskfjdslkfj 2y agoEasier to believe or not, thinking that it's not a reasonable possibility is also funny.
- whimsicalism 2y agoDo you also think they somehow stole the codeforces problems before they were even written or you are willing to believe the #175 global rank there?
- fldskfjdslkfj 2y agoI dont think codeforce claims to contain novel unpublished problems. But i'm not saying it's what they did, just that it's a possibility that should be considered till/if it is debunked.
- deleted 2y ago[deleted]
- roboboffin 2y agoI think the problems it solved were understood to be well known undergraduate problems. https://xenaproject.wordpress.com/2024/12/22/can-ai-do-maths-yet-thoughts-from-a-mathematician/ https://xenaproject.wordpress.com/2024/12/22/can-ai-do-maths...
- deleted 2y ago[deleted]
- whimsicalism 2y agothat is not the correct takeaway from the blogpost at all… 1. it suggests it’s possible that more of the problems are IMO-esque than previously thought, we don’t know how the share of solved problems is. 2. calling IMO problems “well known undergraduate problems” is a bit much
- roboboffin 2y agoThe actual quote is that they are easier than IMO; but maybe not directly comparable. https://x.com/littmath/status/1870848783065788644?s=46&t=foRD58qSBMLXqC-Lq0FCLQ https://x.com/littmath/status/1870848783065788644?s=46&t=foR... I think it’s more probable that it would have solve the easier problems first, rather than some hard and only some easier; although that is supposition. Reading this thread and the blog post gives more idea about what the problems might involve. It’s difficult to judge without more information on the actual results, but that means we cannot draw any strong conclusions either way on what this means.