18 ms·
OpenAI claims gold-medal performance at IMO 2025
- tester756 1y agohuh? any details?
- ktallett 1y ago[flagged]
- littlestymaar 1y agoWhich would be impressive if we knew those problems weren't in the training data already. I mean it is quite impressive how language models are able to mobilize the knowledge they have been trained on, especially since they are able to retrieve information from sources that may be formatted very differently, with completely different problem statement sentences, different variable names and so on, and really operate at the conceptual level. But we must wary of mixing up smart information retrieval with reasoning.
- ktallett 1y agoConsidering these LLM utilise the entirety of the internet, there will be no unique problems that come up in the oLympiad. Even across the course of a degree, you will have likely been exposed to 95% of the various ways to write problems. As you say, retrieval is really the only skill here. There is likely no reasoning.
- colinmorelli 1y agoEven if we accept as a premise that these models are doing "smart retrieval" and not "reasoning" (neither of which are being defined here, nor do I think we can tell from this tweet even if they were), it doesn't really change the impact. There are many industries for which the vast majority of work done is closer to what I think you mean by "smart retrieval" than what I think you mean by "reasoning." Adult primary care and pediatrics, finance, law, veterinary medicine, software engineering, etc. At least half, if not upwards of 80% of the work in each of these fields is effectively pattern matching to a known set of protocols. They absolutely deal in novel problems as well, but it's not the majority of their work. Philosophically it might be interesting to ask what "reasoning" means, and how we can assess if the LLMs are doing it. But, practically, the impacts to society will be felt even if all they are doing is retrieval.
- littlestymaar 1y ago> There are many industries for which the vast majority of work done is closer to what I think you mean by "smart retrieval" than what I think you mean by "reasoning." Adult primary care and pediatrics, finance, law, veterinary medicine, software engineering, etc. At least half, if not upwards of 80% of the work in each of these fields is effectively pattern matching to a known set of protocols. They absolutely deal in novel problems as well, but it's not the majority of their work. I wholeheartedly agree with that. I'm in fact pretty bullish on LLMs, as tools with near infinite industrial use cases, but I really dislike the “AGI soon” narrative (which sets expectations way too high). IMHO the biggest issue with LLMs isn't that they aren't good enough at solving math problem, but that there's no easy way to add information to a model after its training, which is a significant problem for a “smart information retrieval” system. RAG is used as a hack around this issue, but its performance can vary a ton with tasks. LORAs are another options, but they require significant work to make a dataset, and you can only cross your fingers the model keeps its abilities.
- Jcampuzano2 1y agoCalling these high school/early bsc maths questions is an understatement lol.
- demirbey05 1y agoProgress is astounding. Recently report published about evaluation of LLMs on IMO 2025. o3 high didn't even get bronze. https://matharena.ai/imo/ https://matharena.ai/imo/ Waiting for Terry Tao's thoughts, but these kind of things are good use of AI. We need to make science progress faster rather than disrupting our economy without being ready.
- ktallett 1y ago[flagged]
- demirbey05 1y agoI mean progress speed, few months ago they released o3 it has 16 pt in imo 2025
- ktallett 1y agoIn that regards I would agree but that to me suggests that prior hype was unbased though.
- Davidzheng 1y agosorry but I don't think it's accurate to say "they are just challenging for the age range"
- ktallett 1y agoI'm aware you believe they are impossible tasks unless you have specific training, I happen to disagree with that.
- Davidzheng 1y agoyou meaning specific IMO training or general math training? Latter is certainly needed, former being needed in my opinion is a general observation for example about the people who make it on the teams.
- ktallett 1y ago[flagged]
- gametorch 1y ago> high school/early university maths problems should not have been a stretch at all for it This is a ridiculous understatement of the difficulty of getting gold at the IMO.
- ktallett 1y ago[flagged]
- gametorch 1y agoOkay, let's see you try any one of the past IMOs and show us your score. It's really hard. See my other comment. I was voted the best at math in my entire high school by my teachers, completed the first two years of college classes while still in high school. I've tried IMO problems for fun. I'm very happy if I get one right. I'd be infinitely satisfied to score a perfect on 3 out of 6 problems and that's nowhere near gold.
- Davidzheng 1y agoYou'd be surprised at how much math the people who actually get IMO gold know...
- Jcampuzano2 1y agoThere are entire fields of math with exceptional people trying to solve impossibly hard problems that utilize quite literally 0 calculus. Many of them are also questions that eventually end up with proofs or solutions that only require very high level understanding of basic principles. But when I say very high I mean like impossibly high for the average person and ability to combine simple concepts to solve complex problems. I'd wager the majority of Math graduates from universities would struggle to answer most IMO questions.
- oytis 1y ago
- Lionga 1y ago[flagged]
- ktallett 1y ago[flagged]
- baq 1y agoVelocity of AI progress in recent years is exceeded only by velocity of goalposts.
- ktallett 1y agoThe goalposts should focus on being able to make a coherent statement using papers on a subject with sources. At this point it can't do that for any remotely cutting edge topic. This is just a distraction.
- mindwok 1y agoThe idea of a computer being able to solve IMO problems it has not seen before in natural language even just 3 years ago would be completely science fiction. This is astounding progress.
- dang 1y agoPlease see https://news.ycombinator.com/item?id=44617609 https://news.ycombinator.com/item?id=44617609. You degraded this thread badly by posting so many comments like this.
- timbaboon 1y agoHaha no - then it wouldn't have got a gold medal ;)
- z7 1y agoSome previous predictions: In 2021 Paul Christiano wrote he would update from 30% to "50% chance of hard takeoff" if we saw an IMO gold by 2025. He thought there was an 8% chance of this happening. Eliezer Yudkowsky said "at least 16%". Source: https://www.lesswrong.com/posts/sWLLdG6DWJEy3CH7n/imo-challenge-bet-with-eliezer https://www.lesswrong.com/posts/sWLLdG6DWJEy3CH7n/imo-challe...
- exegeist 1y agoImpressive prediction, especially pre-ChatGPT. Compare to Gary Marcus 3 months ago: https://garymarcus.substack.com/p/reports-of-llms-mastering-math-have https://garymarcus.substack.com/p/reports-of-llms-mastering-... We may certainly hope Eliezer's other predictions don't prove so well-calibrated.
- rafaelero 1y agoGary Marcus is so systematically and overconfidently wrong that I wonder why we keep talking about this clown.
- qoez 1y agoPeople just give attention to people making surprising bold counter narrative predictions but don't give them any attention when they're wrong.
- keeda 1y agoPeople like him and Zitron do serve a useful purpose in balancing the hype from the other side, which, while justified to a great extent, is often a bit too overwhelming.
- Philpax 1y agoBeing wrong in the other direction doesn't mean you've found a great balance, it just means you've found a new way to be wrong.
- 1y ago
- orespo 1y agoDefinitely interesting. Two thoughts. First, are the IMO questions somewhat related to other openly available questions online, making it easier for LLMs that are more efficient and better at reasoning to deduce the results from the available content? Second, happy to test it on open math conjectures or by attempting to reprove recent math results.
- ktallett 1y agoYou mean as in the previous years questions will have been used to train it? Yes, they are the same questions and due to them limited format on math questions, there are repeats so LLMs should fundamentally be able to recognise a structure and similarities and use that.
- laurent_du 1y agoThey are not the same question, why are you spreading so much misinformed takes in this thread? I know a guy who had one of the best scores in history at IMO and he's incredibly intelligent. Stop repeating that getting a gold medal at IMO is a piece of cake - it's not.
- samat 1y agoyou either completely misinformed on the topic or a troll
- evrimoztamur 1y agoFrom what I've seen, IMO question sets are very diverse. Moreover, humans also train on all available set of math olympiad questions and similar sets too. It seems fair game to have the AI train on them as well. For 2, there's an army of independent mathematicians right now using automated theorem provers to formalise more or less all mathematics as we know it. It seems like open conjectures are chiefly bounded by a genuine lack of new tools/mathematics.
- dylanbyte 1y agoThese are high school level only in the sense of assumed background knowledge, they are extremely difficult. Professional mathematicians would not get this level of performance, unless they have a background in IMO themselves. This doesn’t mean that the model is better than them in math, just that mathematicians specialize in extending the frontier of math. The answers are not in the training data. This is not a model specialized to IMO problems.
- ktallett 1y ago[flagged]
- Davidzheng 1y agoNo I assure you >50% of working mathematicians will not score gold level at IMO consistently (I'm in the field). As the original parent said, pretty much only ppl who had the training in high school can. Like number theorists without training might be able to do some number theory IMO questions but this level is basically impossible without specialized training (with maybe a few exceptions of very strong mathematicians)
- ktallett 1y agoI sense we may just have a different experience related to colleagues skill sets as I can think of 5 people I could send some questions too and I know they would do them just fine. Infact we often have done similar problems on a free afternoon and I often do similar on flights as a way to pass the time and improve my focus (my issue isn't my talent/understanding at maths, it's my ability to concentrate). I don't disagree that some level of training is needed but these questions aren't unique, nor impossible, especially as said training does exist and LLM's can access said examples. LLM's also have brute force which is a significant help with these type of issues. One particular point is that Math of all the STEM topics to try and focus on probably is the best documented alongside CS.
- Davidzheng 1y ago
- gniv 1y agoFrom that thread: "The model solved P1 through P5; it did not produce a solution for P6." It's interesting that it didn't solve the problem that was by far the hardest for humans too. China, the #1 team got only 21/42 points on it. In most other teams nobody solved it.
- demirbey05 1y agoI think from Canada team someone solved it but among all, its very few
- gus_massa 1y agoIn the IMO, the idea is that the first day you get P1, P2 and P3, and the second day you get P4, P5 and P6. Usually, ordered by difficulty, they are P1, P4, P2, P5, P3, P6. So, usually P1 is "easy" and P6 is very hard. At least that is the intended order, but sometime reality disagree. Edit: Fixed P4 -> P3. Thanks.
- thundergolfer 1y agoYou have P4 twice in there, latter should be 3
- masterjack 1y agoIn this case P6 was unusually hard and P3 was unusually easy https://sugaku.net/content/imo-2025-problems/ https://sugaku.net/content/imo-2025-problems/
- qingcharles 1y agoYikes. 30 years ago I would eat this stuff up and I was the lead dev on 3D engines. Now I can't even make heads-or-tails of what P6 is even asking (＾▽＾)
- tantalor 1y agoThat's very silly. They should do the order like this: Day 1: P1 P3 P5 (odds) Day 2: P2 P4 P6 (evens) Then the problem # is the difficulty.
- deleted 1y ago[deleted]
- davidguetta 1y agoWait for the Chinese version
- johnecheck 1y agoWow. That's an impressive result, but how did they do it? Wei references scaling up test-time compute, so I have to assume they threw a boatload of money at this. I've heard talk of running models in parallel and comparing results - if OpenAI ran this 10000 times in parallel and cherry-picked the best one, this is a lot less exciting. If this is legit, then we need to know what tools were used and how the model used them. I'd bet those are the 'techniques to make them better at hard to verify tasks'.
- Davidzheng 1y agoI don't think it's much less exciting if they ran it 10000 parallel? It implies an ability to discern when the proof is correct and rigorous (which o3 can't do consistently) and also means that outputting the full proof is within capabilities even if rare.
- FeepingCreature 1y agoThe whole point of RL is if you can get it to work 0.01% of the time you can get it to work 100% of the time.
- lcnPylGDnU4H9OF 1y ago> what tools were used and how the model used them According to the twitter thread, the model was not given access to tools.
- constantcrying 1y ago>if OpenAI ran this 10000 times in parallel and cherry-picked the best one, this is a lot less exciting. That entirely depends on who did the cherry picking. If the LLM had 10000 attempts and each time a human had to falsify it, this story means absolutely nothing. If the LLM itself did the cherry picking, then this is just akin to a human solving a hard problem. Attempting solutions and falsifying them until the desired result is achieved. Just that the LLM scales with compute, while humans operate only sequentially.
- 1y ago
- deleted 1y ago[deleted]
- reactordev 1y agoThe Final boss was: Which is greater, 9.11 or 9.9? /s I kid, this is actually pretty amazing!! I've noticed over the last several months that I've had to correct it less and less when dealing with advanced math topics so this aligns.
- zkmon 1y agoThis is an awesome progress in human achievement to get these machines intelligent. And this is also a fast regress and decline on the human wisdom! We are simply greasing the grooves and letting things slide faster and faster and calling it progress. How does this help to make the human and nature integration better? Does this improve climate or make humans adapt better to changing climate? Are the intelligent machines a burning need for the humanity today? Or is it all about business and political dominance? At what cost? What's the fall out of all this?
- jebarker 1y agoNobody knows the answers to these questions. Relying on AGI solving problems like climate change seems like a risky strategy but on the other hand it’s very plausible that these tools can help in some capacity. So we have to build, study and find out but also consider any opportunity cost of building these tools versus others.
- jfengel 1y agoSolving climate change isn't a technical problem, but a human one. We know the steps we have to take, and have for many years. The hard part is getting people to actually do them. No human has any idea how to accomplish that. If a machine could, we would all have much to learn from it.
- jebarker 1y agoI disagree with this assessment. We don’t know the steps we have to take. We know a set of steps we could take but they’re societally unpalatable. Technology can potentially offer alternative steps or introduce societal changes that make the first set of steps more palatable.
- jebarker 1y agoI feel I should clarify as clearly this is an unpopular opinion: I’m not saying climate change can be solved by technology alone, but I do believe that enabling the societal changes needed to deal with climate change requires using every tool we have at our disposal and that includes technology. I don’t really see why this is controversial and would love to hear that perspective.
- chairhairair 1y agoOpenAI simply can’t be trusted on any benchmarks: https://news.ycombinator.com/item?id=42761648 https://news.ycombinator.com/item?id=42761648
- amelius 1y agoThis is not a benchmark, really. It's an official test.
- andrepd 1y agoAnd what were the methods? How was the evaluation? They could be making it all up for all we know!
- PokemonNoGo 1y agoWhat is an _official_ test?
- qoez 1y agoRemember that they've fired all whistleblowers that would admit to breaking the verbal agreement that they wouldn't train on the test data.
- samat 1y agoCould not find it on the open web. Do you have clues to search for?
- deleted 1y ago[deleted]
- Aurornis 1y agoThe International Math Olympiad isn’t an AI benchmark. It’s an annual human competition.
- meroes 1y agoThey didn’t actually compete.
- tlb 1y agoI encourage anyone who thinks these are easy high-school problems to try to solve some. They're published (including this year's) at https://www.imo-official.org/problems.aspx https://www.imo-official.org/problems.aspx. They make my head spin.
- xpressvideoz 1y agoI didn't know there were localized versions of the IMO problems. But now that I think of it, having versions of multiple languages is a must to remove the language barrier from the competitors. I guess having that many language versions (I see ~50 languages?) may make keeping the security of the problems considerably harder?
- dmurray 1y agoThe problems are chosen by representatives from all the countries. So every country has someone who knows the full exam before the participants get it. Security is on the honour system, but it seems to mostly work.
- peteyreplies 1y agoiirc, the IMO system automatically translates the questions into 50 languages, after they are entered in English.
- kappi 1y ago[flagged]
- throw310822 1y agoI think you're joking, but you never know :)
- yongjik 1y agoAll the past IMO problems are known to public and contestants practice on them. If solving an IMO problem is the simple matter of "looking at all the past problems and apply the same pattern," you'd expect human contestants to do a lot better.
- ksec 1y agoI am neither an optimist nor a pessimist for AI. I would likely be called both by the opposite parties. But the fact that AI / LLM is still rapidly improving is impressive in itself and worth celebrating for. Is it perfect, AGI, ASI? No. Is it useless? Absolutely not. I am just happy the prize is so big for AI that there are enough money involve to push for all the hardware advancement. Foundry, Packaging, Interconnect, Network etc, all the hardware research and tech improvements previously thought were too expensive are now in the "Shut up and take my money" scenario.
- asadotzler 1y agoBut unlike the trillion dollars invested in the broadband internet build out between 1998 and 2008, when this 10 year trillion dollar bubble pops, we won't be left with an enduring and useful piece of infrastructure adding a trillion dollars to the global economy annually.
- jononor 1y agoIt would leave a lots of general purpose GPU-based compute. That is useful and enduring infrastructure? These things are used for many scientific and engineering problems - including medicine, climate modeling, material science, neuroscience, etc
- TacticalCoder 1y ago> ... we won't be left with an enduring and useful piece of infrastructure adding a trillion dollars to the global economy annually. I'm not drinking the AGI kool-aid but I use LLMs daily. We pay not one but two AI subscriptions at home (including Claude). It's extremely useful. From translation to proof-reading to synthetizing to expanding on something to writing little dumb functions to helping with spreadsheet formulas to documenting code to writing commit messages to helping find movie names (when I only remember very partially the plot) etc. How is this not already adding a trillion dollars to the economy? It's not about the infrastructure: all that counts are the models. They're here to stay. They're not going away. It's the single biggest time-saver I've ever seen for mundane tasks (and, no, it doesn't write good code: it write shitty pathetic underperforming insecure code... And yet it's still useful for proofs of concept / one-offs / throwaway).
- andrepd 1y agoAm I missing something or is this completely meaningless? It's 100% opaque, no details whatsoever and no transparency or reproducibility. I wouldn't trust these results as it is. Considering that there are trillions of dollars on the line as a reward for hyping up LLMs, I trust it even less.
- flappyeagle 1y agoYes you are missing the entire boat
- aerhardt 1y agoThe entire boat is hidden, we see nothing but a projected shadow, but we are to be blamed for missing it?
- meroes 1y agoIn the RLHF sphere you could tell some AI company/companies were targeting this because of how many IMO RLHF’ers they were hiring specifically. I don’t think it’s really easy to say how much “progress” this is given that.
- ainch 1y agoI doubt this is coming from RLHF - tweets from the lead researcher state that this result flows from a research breakthrough which enables RLVR on less verifiable domains.
- meroes 1y agoMath RLHF already has verifiable ground truth/right vs wrong, so I don't what this distinction really shows. And AI changes so quickly that there is a breakthrough every week. Call my cynical, but I think this is an RLHF/RLVR push in a narrow area--IMO was chosen as a target and they hired specifically to beat this "artificial" target.
- Davidzheng 1y agoRLHF means Reinforcement Learning from Human Feedback. The right/wrong ones are either called RL or RLVR (Verfiable Rewards)
- saagarjha 1y agoThey were hiring IMO winners because IMO winners tend to be good at working on AI, not because they had the people specifically to make the AI better at math.
- meroes 1y agoUh no. I’m a math RLHF’er. When I get hired, I work on math/logic up to masters level because that’s my qualifications. Masters and PHD work on masters and PHD level. And IMO work on IMO math. Every skill and skill level is specifically assigned and hired in the RLHF world. Sometime the skill levels are fuzzier, but that’s usually very temporary. And as been said already, IMO is a specific skill that even PHD math holders aren’t universally trained for.
- ALLTaken 1y ago[flagged]
- deleted 1y ago[deleted]
- aubanel 1y ago- AI competing is "wholly unfair" - "[AI is] far away from being substantially being better than MCTs" ^ pick only one
- stingraycharles 1y agoYeah it’s a completely fair playing field, it’s completely obvious that AI should be able to compete with humans in the same way that robotics and computers can compete with humanity (and are better suited for many tasks). Whether or not they’re far away from being better than humans is up to debate, but the entire point of these types of benchmarks it to compare them to humans.
- bluecalm 1y ago>>Yeah it’s a completely fair playing field, it’s completely obvious that AI should be able to compete with humans in the same way that robotics and computers can compete with humanity (and are better suited for many tasks). Yeah same way computers and robots should be able to win World Chess Championship, 100m dash and Wimbledon. >>but the entire point of these types of benchmarks it to compare them to humans The entire point of the competition is to fight against participants who are similar to you, have similar capabilities and go through similar struggles. If you want bot vs human competitions - great - organize it yourself instead of hijacking well established competitions out there.
- yobbo 1y agoRunning MCTS over algorithms is the part that might be considered unfair if used in competition with humans.
- up2isomorphism 1y agoIn fact no car company claims “gold medal” performance in Olympic running even they can do that 100 yeas ago. Obviously since IMO does not generate much money so it is an easy target. BTW; “Gold medal performance “ looks a promotional term for me.
- ddtaylor 1y agoGlock should show up to the UFC and win the whole tournament handily.
- up2isomorphism 1y agoExactly!
- flappyeagle 1y agoLMAO
- amelius 1y agoMakes sense. Mathematicians use intuiton a lot to drive their solution seeking, and I suppose an AI such as an LLM could develop intuition too. Of course where AI really wins is search speed and the fact that an LLM really doesn't get tired when exploring different strategies and steps within each strategy. However, I expect that geometric intuition may still be lacking mostly because of the difficulty of encoding it in a form which an LLM can easily work with. After all, Chatgpt still can't draw a unicorn [1] although it seems to be getting closer. [1] https://gpt-unicorn.adamkdean.co.uk/ https://gpt-unicorn.adamkdean.co.uk/
- quirino 1y agoI think equally impressive is the performance of the OpenAI team at the "AtCoder World Tour Finals 2025" a couple of days ago. There were 12 human participants and only one did better than OpenAI. Not sure there is a good writeup about it yet but here is the livestream: https://www.youtube.com/live/TG3ChQH61vE https://www.youtube.com/live/TG3ChQH61vE.
- zeroonetwothree 1y agoAnd yet when working on production code current LLMs are about as good as a poor intern. Not sure why the disconnect.
- kenjackson 1y agoDepends. I’ve been using it for some of my workflows and I’d say it is more like a solid junior developer with weird quirks where it makes stupid mistakes and other times behaves as a 30 year SME vet.
- Rioghasarig 1y agoI really doubt it's like a "solid junior developer". If it could do the work of a solid junior developer it would be making programming projects 10-100x faster because it can do things several times faster than a person can. Maybe it can write solid code for certain tasks but that's not the same thing as being a junior developer.
- peripitea 1y agoIt can be 10-100x faster for some tasks already. I've had it build prototypes in minutes that would have taken me a few hours to cobble together, especially in domains and using libraries I don't have experience with.
- riku_iki 1y agobecause competitive coding is narrow well described domain(limited number of concepts: lists, trees, etc) with high volume of data available for training, and easy way to setup RL feeback loop, so models can improve well in this domain, which is not true about typical enterprise overbloated software.
- chvid 1y agoI believe this company used to present its results and approach in academic papers with enough details so that it could be reproduced by third parties. Now it is just doing a bunch of tweets?
- samat 1y agoThis company used to be non profit And many other things
- do_not_redeem 1y ago[flagged]
- samrus 1y agothats when they were a real research company. the last proper research they did was instructGPT, everything since has been product development and following others. the reputation hit hasnt caught up with them because sam altman has built a whole career out of outrunning the reputation lag
- modeless 1y agoNoam Brown: > this isn’t an IMO-specific model. It’s a reasoning LLM that incorporates new experimental general-purpose techniques. > it’s also more efficient [than o1 or o3] with its thinking. And there’s a lot of room to push the test-time compute and efficiency further. > As fast as recent AI progress has been, I fully expect the trend to continue. Importantly, I think we’re close to AI substantially contributing to scientific discovery. I thought progress might be slowing down, but this is clear evidence to the contrary. Not the result itself, but the claims that it is a fully general model and has a clear path to improved efficiency. https://x.com/polynoamial/status/1946478249187377206 https://x.com/polynoamial/status/1946478249187377206
- YeGoblynQueenne 1y agoHow is a claim, "clear evidence" to anything?
- kelipso 1y ago[flagged]
- modeless 1y agoMost evidence you have about the world is claims from other people, not direct experiment. There seems to be a thought-terminating cliche here on HN, dismissing any claim from employees of large tech companies. Unlike seemingly most here on HN, I judge people's trustworthiness individually and not solely by the organization they belong to. Noam Brown is a well known researcher in the field and I see no reason to doubt these claims other than a vague distrust of OpenAI or big tech employees generally which I reject.
- stingraycharles 1y agoMy issue with all these citations is that it’s all OpenAI employees that make these claims. I’ll wait to see third party verification and/or use it myself before judging. There’s a lot of incentives right now to hype things up for OpenAI.
- do_not_redeem 1y agoA third party tried this experiment with publicly available models. OpenAI did half as well as Gemini, and none of the models even got bronze. https://matharena.ai/imo/ https://matharena.ai/imo/
- jsnell 1y agoI feel you're misunderstanding something. That's not "this exact experiment". Matharena is testing publicly available models against the IMO problem set. OpenAI was announcing the results of a new, unpublished model, on that problems set. It is totally fair to discount OpenAI's statement until we have way more details about their setup, and maybe even until there is some level of public access to the model. But you're doing something very different: implying that their results are fraudulent and (incorrectly) using the Matharena results as your proof.
- do_not_redeem 1y agoFair enough, edited.
- gettingoverit 1y agoImplying results are fraudulent is completely fair when it is a fraud. The previous time they had claims about solving all of the math right there and right then, they were caught owning the company that makes that independent test, and could neither admit nor deny training on closed test set.
- tedsanders 1y agoJust to quickly clarify: - OpenAI doesn't own Epoch AI (though they did commission Epoch to make the eval) - OpenAI denied training on the test set (and further denied training on FrontierMath-derived data, training on data targeting FrontierMath specifically, or using the eval to pick a model checkpoint; in fact, they only downloaded the FrontierMath data after their o3 training set was frozen and they didn't look at o3's FrontierMath results until after the final o3 model was already selected. primary source: https://x.com/__nmca__/status/1882563755806281986 https://x.com/__nmca__/status/1882563755806281986) You can of course accuse OpenAI of lying or being fraudulent, and if that's how you feel there's probably not much I can say to change your mind. One piece of evidence against this is that the primary source linked above no longer works at OpenAI, and hasn't chosen to blow the whistle on the supposed fraud. I work at OpenAI myself, training reasoning models and running evals, and I can vouch that I have no knowledge or hints of any cheating; if I did, I'd probably quit on the spot and absolutely wouldn't be writing this comment. Totally fine not to take every company's word at face value, but imo this would be a weird conspiracy for OpenAI, with very high costs on reputation and morale.
- YeGoblynQueenne 1y agoGuys, that's nothing. My new AI system is not LLM-based but neuro-symbolic and yet it just scored 100% on the IMO 2026 problems that haven't even been written yet, it is that good. What? This is a claim with all the trust-worthiness of OpenAI's claim. I mean I can claim anything I want at this point and it would still be just as trust-worthy as OpenAI's claim, with exactly zero details about anything else than "we did it, promise".
- Jackson__ 1y agoAlso interesting takeaways from that tweet chain: >GPT5 soon >it will not be as good as this secret(?) model
- mehulashah 1y agoThe AI scaling that went on for the last five years is going to be very different from the scaling that will happen in the next ten years. These models have latent capabilities that we are racing to unearth. IMO is but one example. There’s so much to do at inference time. This result could not have been achieved without the substrate of general models. Its not like Go or protein folding. You need the collective public global knowledge of society to build on. And yes, there’s enough left for ten years of exploration. More importantly, the stakes are high. There may be zero day attacks, biological weapons, and more that could be discovered. The race is on.
- mikert89 1y agoYup, we have bootstrapped to enough intelligence in the models that we can introduce higher levels of ai
- meroes 1y agoLatent?? If you looked at RLHF hiring over the last year, there was a huge hiring of IMO competitors to RLHF. This was a new, highly targeted, highly funded RLHF’ing.
- curiousguy7374 1y agoCan you provide any kind of source? Very curious about this!
- meroes 1y agohttps://work.mercor.com/jobs/list_AAABljpKHPMmFMXrg2VM0qz4 https://work.mercor.com/jobs/list_AAABljpKHPMmFMXrg2VM0qz4 https://benture.io/job/international-math-olympiad-participant-for-ai-at-mercor https://benture.io/job/international-math-olympiad-participa... https://job-boards.greenhouse.io/xai/jobs/4538773007 https://job-boards.greenhouse.io/xai/jobs/4538773007 And Outlier/Scale, which was bought by Meta (via Scale), had many IMO-required Math AI trainer jobs on LinkedIn. I can't find those historical ones though. I'm just one piece in the cog and this is an anecdote, but there was a huge upswing in IMO or similar RLHF job postings over the past 6mo-year.
- mikert89 1y agoThe cynicism/denial on HN about AI is exhausting. Half the comments are some weird form of explaining away the ever increasing performance of these models I've been reading this website for probably 15 years, its never been this bad. many threads are completely unreadable, all the actual educated takes are on X, its almost like there was a talent drain
- softwaredoug 1y agoProbably because both sides have strong vested interests and it’s next to impossible to find a dispassionate point of view. The Pro AI crowd, VC, tech CEOs etc have strong incentive to claim humans are obsolete. Many tech employees see threats to their jobs and want to poopoo any way AI could be useful or competitive.
- orbital-decay 1y agoThat's a huge hyperbole. I can assure you many people find the entire thing genuinely fascinating, without having any vested interest and without buying the hype.
- spacemadness 1y agoSure but it’s still a gold rush with a lot of exaggeration pushed by tech executives to acquire investors. There’s a lot of greed and fear to go around. I think LLMs are fascinating and cool myself having grown up with Eliza and crappy expert systems, but am more interested in deep learning outcomes like Alphafold than general purpose LLMs. You don’t hear enough about non-LLM AI because of all the money riding on LLM based tech. It’s hard not to see the bad behavior that has arisen due to all the money being thrown about. So that is to say it makes sense there is some skepticism as you can’t take what these companies say at face value. It’d be nice to have a toned down discussion about what LLMs can and can’t do but there is a lot of obfuscation and hype. Also there is the conversation about what they should or shouldn’t be doing which is completely fair to talk about.
- rvz 1y ago
- csomar 1y agoThe issue is that trust is very hard to build and very easy to lose. Even in today's age where regular humans have a memory span shorter than that of an LLM, OpenAI keeps abusing the public's trust. As a result, I take their word on AI/LLMs about as seriously as I'd take my grocery store clerk's opinion on quantum physics.
- emp17344 1y agoI still haven’t forgotten OpenAI’s FrontierMath debacle from December. If they really have some amazing math-solving model, give us more info than a vague twitter hype-post.
- deleted 1y ago[deleted]
- riku_iki 1y ago> The issue is that trust is very hard to build and very easy to lose I think its opposite: general public blindly trusts all kind of hyped stuff, its a very few hyper-skeptical who are some fraction of percent of population.
- gniv 1y agoEspecially since they are saying they don't plan to release this kind of model anytime soon.
- gcanyon 1y ago99.99+% of all problems humans face do not require particularly original solutions. Determining whether LLMs can solve truly original (or at least obscure) problems is interesting, and a problem worth solving, but ignores the vast majority of the (near-term at least) impact they will have.
- another_twist 1y agoIts a level playing field IMO. But theres another thread which claims not even bronze and I really don't want to go to X for anything.
- Davidzheng 1y agoI can save you the click. Public models (gemini/o3) are less than bronze. this is a specially trained model which is not publicly available.
- aerhardt 1y agoIs the conclusion that elite models are being withheld from the public, or that general models are not that general?
- famouswaffles 1y agoelite(er) models are always being withheld from the public. Open AI sat on GPT-4 for 8 months before actually releasing it.
- tootyskooty 1y agoThe conclusion is that research takes time to productize, and this is cutting-edge research. OAI employees stated that there isn't anything math-specific (think AlphaGeometry) about this model. It's a general system.
- another_twist 1y agoI am quite surprised that Deepmind with MCTS wasnt able to figure out math performance itself.
- tootyskooty 1y agoGoogle will also have good results to report for this year's IMO, OpenAI just beat them to the announcement
- NitpickLawyer 1y agoI think google did some official collaboration with IMO, and will announce later. Or at least that's what I read from the IMO official saying "AI companies should wait 1 week before announcing so that we can celebrate the human winners" and "to my knowledge oai was not officially collaborating with IMO" ...
- rafael859 1y agoInteresting that the proofs seem to use a limited vocabulary: https://github.com/aw31/openai-imo-2025-proofs/blob/main/problem_1.txt https://github.com/aw31/openai-imo-2025-proofs/blob/main/pro... Why waste time say lot word when few word do trick :) Also worth pointing out that Alex Wei is himself a gold medalist at IOI.
- NitpickLawyer 1y ago> Also worth pointing out that Alex Wei is himself a gold medalist at IOI. Terence Tao also called it, that the top LLMs would get gold this year in a recent podcast.
- deleted 1y ago[deleted]
- johnpaulkiser 1y agoAre you saying "see the world?" or "seaworld"?
- deleted 1y ago[deleted]
- beyonddream 1y agoHe is talking about IMO (math olympiad) while he got gold at IOI (informatics olympiad) :)
- throw310822 1y agoInteresting observation. One one hand, these resemble more the notes that an actual participant would write while solving the problem. Also, less words = less noise, more focus. But also, specifically for LLMs that output one token at a time and have a limited token context, I wonder if limiting itself to semantically meaningful tokens can be create longer stretches of semantically coherent thought?
- 1y ago
- darkoob12 1y agoI don't know how much novelty should you expect from IMO every year but i expect many of them be variation of the same problem. These models are trained on all old problem and their various solutions.For LLM models, solving thses problems are as impressive as writing code. There is no high generalization.
- mathluke 1y agoYou should expect quite a bit of novelty from the IMO, given the constraint of high school level curriculum. The problem setters work very hard to avoid problems that are variations of other contests or solvable by routine methods. That's why this is a very exciting result--you can't just regurgitate homework problem solutions to get a high score at the IMO.
- bluecalm 1y agoMy view is that it's less impressive than previous go and chess results. Humans are worse at competitive math than at those games, it's still very limited space and well defined problems. They may hype "general purpose" as much as they want but for now it's still the case that AI is super human at well defined limited space tasks and can't achieve performance of a mediocre below average human at simple tasks without those limitations like driving a car. Nice result but it's just another game humans got beaten at. This time a game which isn't even taken very seriously (in comparison to ones that have professional scene).
- azan_ 1y agoThe scope and creativity required for IMO is much bigger than chess/GO. Also IMO is taken VERY seriously. It's a huge deal, much bigger than any chess or go tournaments.
- bluecalm 1y agoImo competitive math (or programming) is about knowing some tricks and then trying to find a combination of them that works for a given task. The number of tricks and depth required is much less than in go or chess. I don't think it's very creative endeavor in comparison to chess/go. The searching required is less as well. There is a challenge processing natural language and producing solutions in it though. Creativity required is not even a small fraction of what is required for scientific breakthroughs. After all no task that you can solve in 30 minutes or so can possibly require that much creativity - just knowledge and a fast mind - things computers are amazing at. I am AI enthusiast. I just think a lot of things that were done so far are more impressive than being good at competitive math. It's a nice result blown out of proportion by OpenAI employees.
- mathluke 1y agoI'd disagree with this take. Math olympiads are some of the most intellectually creative activities I've ever done that fit within a one day time limit. Chess and go don't even come close--I am not a strong player, but I've studied both games for hundreds of hours. (My hot take is that chess is not even very creative at all, that's why classical AI techniques produced super human results many years ago.) There is no list of tricks that will get a silver much less a gold medal at the IMO. The problem setters try very hard to choose problems that are not just variations of other contests or solvable by routine calculation (indeed some types of problems, like polynomial inequalities, fell out of favor as near-universal techniques made them too routine to well prepared students). Of course there are common themes and patterns that recur--no way around it given the limited curriculum they draw on--but overall I think the IMO does a commendable job at encouraging out-of-the-box thinking within a limited domain. (I've heard a contestant say that IMO prep was memorizing a lot of template solutions, but he was such a genius among geniuses that I think his opinion is irrelevant to the rest of humanity!) Of course there is always a debate whether competition math reflects skill in research math and other research domains. There's obvious areas of overlap and obvious areas of differences, so it's hard to extrapolate from AI math benchmarks to other domains. But i think it's fair to say the skills needed for the IMO include quite general quantitative reasoning ability, which is very exciting to see LLMs develop.
- deleted 1y ago[deleted]
- mhh__ 1y agoJust a stochastic parrot bro
- TesterVetter 1y ago[dead]
- strangeloops85 1y agoIt’s interesting how hard and widespread a push they’re making in advertising this - at this particular moment, when there are rumors of more high level recruitment attempts / successes by Zuck. OpenAI is certainly a master at trying to drive narratives. (Independent of the actual significance / advance here). Sorry, there are too many hundreds of billions of dollars involved to not be a bit cautious and wary of claims being pushed this hard.
- Philpax 1y agoOfficial OpenAI announcements: https://xcancel.com/OpenAI/status/1946594928945148246 https://xcancel.com/OpenAI/status/1946594928945148246 https://xcancel.com/OpenAI/status/1946594933470900631 https://xcancel.com/OpenAI/status/1946594933470900631
- mrdependable 1y agoI like how they always say AI will advance science when they want to sell it to the public, but pump how it will replace workers when selling it to businesses. It’s like dangling a carrot while slowly putting a knife to our throats. Edit: why was my comment moved from the one I was replying to? It makes no sense here on its own.
- eab- 1y ago[dead]
- riku_iki 1y ago> such announcements should wait at least a week after the closing ceremony it would raise more concerns that corps leaked questions/answers to training data and finetuned specialized models during this time.
- demirbey05 1y agoGoogle also joined IMO, and got gold prize. https://x.com/natolambert/status/1946569475396120653 https://x.com/natolambert/status/1946569475396120653 OAI announced early, probably we will hear announcement from Google soon.
- futureshock 1y agoGoogle’s AlphaProof, which got a silver last year, has been using a neural symbolic approach. This gold from OpenAI was pure LLM. We’ll have to see what Google announces, but the LLM approach is interesting because it will likely generalize to all kinds of reasoning problems, not just mathematical proofs.
- skepticATX 1y agoOpenAI’s systems haven’t been pure language models since the o models though, right? Their RL approach may very well still generalize, but it’s not just a big pre-trained model that is one-shotting these problems. The key difference is that they claim to have not used any verifiers.
- beering 1y agoWhat do you mean by “pure language model”? The reasoning step is still just the LLM spitting out tokens and this was confirmed by Deepseek replicating the o models. There’s not also a proof verifier or something similar running alongside it according to the openai researchers. If you mean pure as in there’s not additional training beyond the pretraining, I don’t think any model has been pure since gpt-3.5.
- gallerdude 1y agoLocal models you can get just the pretrained versions of, no RLHF. IIRC both Llama and Gemma make them available.
- alach11 1y ago> it will likely generalize to all kinds of reasoning problems, not just mathematical proofs Big if true. Setting up an RL loop for training on math problems seems significantly easier than many other reasoning domains. Much easier to verify correctness of a proof than to verify correctness (what would this even mean?) for a short story.
- skepticATX 1y agoHas anyone independently reviewed these solutions? My proving skills are extremely rusty so I can’t look at these and validate them. They certainly are not traditional proofs though.
- shiandow 1y agoI read through P1, and it seemed to be correct. Though you could explain the central idea of the proof into about 3 sentences and a few drawings. It reads like someone who found the correct answer but seemingly had no understanding of what they did and just handed in the draft paper. Which seems odd, shouldn't an LLM be better at prose?
- matt123456789 1y agoOne would think. I suppose OpenAI threw the majority of their compute budget at producing and verifying solutions. It would certainly be interesting to see whether or not this new model can distill its responses to just those steps necessary to convey its result to a given audience.
- PokemonNoGo 1y ago> level performance on the world’s most prestigious math competition I don't know which one i would consider the most prestigious math competition but it wouldn't be The IMO. The Putnam ranks higher to me and I'm not even an American. But I've come to realise one thing and that is that high-school is very important to Americans...
- sealeck 1y agoThe Putnam and IMO are quite different. I would suggest the IMO is probably harder...
- kevinventullo 1y agoI would disagree; the IMO depends only on late middle school/early high school level mathematics (geometry, gcd, functions) while Putnam typically depends on late high school/early college-level mathematics (integrals, limits, matrices).
- sigbottle 1y agoWell I come from the competitive programming sphere, and I would say that IOI is harder than ICPC. When you don't know that many things, that's when creativity shines, and there are some truly genuinely shocking IOI problems. ICPC (well, in recent years they've gotten slightly better) is pretty well known as a knowledge-heavy implementation contest. Many teams get the experience that they mind-solved a lot more problems but couldn't implement them in time. Typing up the maxflow template for the 25th time for a series of collegiate-level but ultimately standard reductions isn't that inspiring. My favorite problems are those you can derive from basic techniques but come up with scaffolding that is truly elegant. I've set some of them myself, which have stumped some famous people you may know :) --- I guess my point is that I can see people feeling about Putnam the same way.
- kurtis_reed 1y agoThank you for this invaluable contribution
- pradn 1y ago> Btw, we are releasing GPT-5 soon, and we’re excited for you to try it. But just to be clear: the IMO gold LLM is an experimental research model. We don’t plan to release anything with this level of math capability for several months. GPT-5 finally on the horizon!
- nmca 1y agoIt’s interesting that this is a competition elite enough that several posters on a programming website don’t seem to understand what it is. My very rough napkin math suggests that against the US reference class, imo gold is literally a one in a million talent (very roughly 20 people who make camp could get gold out of very roughly twenty million relevant high schoolers).
- skepticATX 1y agoI’m not trying to take away from the difficulty of the competition. But I went to a relatively well regarded high school and never even heard of IMO until I met competitors during undergrad. I think that the number of students who are even aware of the competition is way lower than the total number of students. I mean, I don’t think I’d have been a great competitor even if I tried. But I’m pretty sure there are a lot of students that could do well if given the opportunity.
- SJC_Hacker 1y agoAre you in the US? Have you heard of the AMC (used to be AHMSE) and the AIME? Those are the feeders to the IMO. If your school had a math team and you were on it, would be surprised if you didn't hear of it You may not have heard of the IMO because no one in school district, possibly even state got in. It is extremely selective (like 20 students in the entire country)
- skepticATX 1y agoI’m in the US but this was a while back, in the south. It was a highly ranked school and ended up producing lots of PhDs, but many of the families were blue collar and so there just wasn’t any awareness of things like this.
- ajennings 1y agoI'm curious on your response to GP's question. Have you heard of AHSME, AMC, or AIME? Nobody mentioned them in high school (1997) until I heard of them online and got my school to participate. 30 kids took the AHSME. Only one qualified for the AIME. And nobody qualified for IMO (though I tell myself I was close). I believe the 1 in a million number.
- neonate 1y agohttps://xcancel.com/alexwei_/status/1946477742855532918 https://xcancel.com/alexwei_/status/1946477742855532918 https://xcancel.com/polynoamial/status/1946478249187377206 https://xcancel.com/polynoamial/status/1946478249187377206
- sorokod 1y agoNot even bronze. https://news.ycombinator.com/item?id=44615695 https://news.ycombinator.com/item?id=44615695
- Philpax 1y agoThis is a new model. Those tests were with what's currently publicly available.
- sorokod 1y agoAh true. Although it's the benchmark that is publicly available. The model is not.
- amelius 1y agoFrom the article: > Btw, we are releasing GPT-5 soon, and we’re excited for you to try it. But just to be clear: the IMO gold LLM is an experimental research model. We don’t plan to release anything with this level of math capability for several months.
- neets 1y agoSolve aging and disease when?
- strangescript 1y agoWe are the frog in the warm water...
- archon1410 1y agoAnd of course it's available even in Icelandic, spoken by ~300k people, but not a single Indian language, spoken by hundreds of millions. भारत दुर्दशा न देखी जाई...
- sebzim4500 1y agoPresumably almost all competitors from India would be fluent in English (given it is the second most spoken language there)? I guess the same is true of Icelandic though.
- archon1410 1y agoYes, and there's also languages of ex-USSR countries, whose competitors presumably all understand Russian, and so on. The real reason might be that there's an enormous class of self-loathing elites in India who actively despise the possibility of any Indian language being represented in higher education. This obviously stunts the possibility of them being used in international competitions.
- tmule 1y agoDiscussions about Indian politics or the Indian psyche—especially when laced with Indic supremacist undertones—are off-topic and an annoyance here. Please consider sharing these views in a forum focused on Indian affairs, where they’re more likely to find the traction they deserve.
- archon1410 1y agoIt is not "supremacist" to believe that depriving hundreds of millions of people from higher education in their native language is deeply unjust. This reflection was prompted by a comment on why Indian languages are not represented in international competitions, which was prompted by a comment on the competition being available in many languages. Discussions online have a tendency to go off into tangents like this. It's regrettable that this is such a contentious topic.
- mohsen1 1y agoO3 Pro could solve and prove the first problem when I tried: https://chatgpt.com/s/t_687be6c1c1b88191b10bfa7eb1f37c07 https://chatgpt.com/s/t_687be6c1c1b88191b10bfa7eb1f37c07
- bwfan123 1y agoI fed the problem 1 solution into gemini and asked if it was generated by a human or llm. It said: Conclusion: It is overwhelmingly likely that this document was generated by a human. ---- Self-Correction/Refinement and Explicit Goals: "Exactly forbidden directions. Good." - This self-affirmation is very human. "Need contradiction for n>=4." - Clearly stating the goal of a sub-proof. "So far." - A common human colloquialism in working through a problem. "Exactly lemma. Good." - Another self-affirmation. "So main task now: compute K_3. And also show 0,1,3 achievable all n. Then done." - This is a meta-level summary of the remaining work, typical of human problem-solving " ----
- bigyabai 1y agoLLMs are not an accurate test of whether something was written by an LLM or not.
- matt3210 1y agoIf stealing content to train models is ok, then is stealing models to merge ok?
- atleastoptimal 1y ago>AI model performs astounding feat everyone claimed was impossible or won’t be achieved for a while >Commenters on HN claim it must not be that hard, or OpenAI is lying, or cheated. Anything but admit that it is impressive Every time on this site lol. A lot of people here have an emotional aversion to accepting AI progress. They’re deep in the bargaining/anger/denial phase.
- demaga 1y agoIt's a good camp to be in. If we're wrong, that will be awesome!
- Oras 1y agoGet ready to downvoted, the wave of single minded is coming
- atleastoptimal 1y agoThere is a diversity of opinions on this site. I do hope that soon more of the intelligent commenters who have spent a while denying AI progress will realize what’s actually happening and contribute their brainpower to a meaningful cause in the lead up to AI-human parity. If we want a good future in a world where AI is smarter than humans, we need to do alignment work soon.
- tomrod 1y agoSort of a naive response, considering many of the folks calling out the issues have significant experience building with LLMs or building LLMs.
- OldfieldFund 1y agoI'm building with LLMs, and they're solving problems that weren't possible to solve before due to how many resources they would consume. Resources, as in human-hours. Finance, chemistry, biology, medicine.
- atleastoptimal 1y ago
- OtherShrezzing 1y agoThe 4 hours time limit is a bit of an odd metric, given that OpenAI have effectively an unlimited amount of compute at their disposal. If they’re running the model on 100,000 GPUs for 4hrs, that’s obviously going to have better outcomes than running it on 5.
- cowpig 1y agohttps://threadreaderapp.com/thread/1946477742855532918.html https://threadreaderapp.com/thread/1946477742855532918.html
- charlieyu1 1y agoI tried P1 on chatgpt-o4-high, it tells me the solution is k=0 or 1. It doesn’t even know that k=3 is a solution for n=3. Such a solution would get 0/7 in the actual IMO.
- CamperBob2 1y agoWhat about o3-pro? Remember that their model names make no sense. Edit due to rate-limiting: o3-pro returned an answer after 24 minutes: https://chatgpt.com/share/687bf8bf-c1b0-800b-b316-ca7dd9b00941 https://chatgpt.com/share/687bf8bf-c1b0-800b-b316-ca7dd9b009... Whether the CoT amounts to valid mathematical reasoning, I couldn't say, especially because OpenAI models tend to be very cagey with their CoT. Gemini 2.5 Pro seems to have used more sophisticated reasoning ( https://g.co/gemini/share/c325915b5583 https://g.co/gemini/share/c325915b5583 ) but it got a slightly different answer. Its chain of thought was unimpressive to say the least, so I'm not sure how it got its act together for the final explanation. Claude Opus 4 appears to have missed the main solution the others found: https://claude.ai/share/3ba55811-8347-4637-a5f0-fd8790aa820b https://claude.ai/share/3ba55811-8347-4637-a5f0-fd8790aa820b Be interesting if someone could try Grok 4.
- charlieyu1 1y agoOnly have basic o3 to try. Spent like 10 minutes but did not return any response due to a network error. Checking the thoughts, the model was doing a lot of brute forcing up to n=8, and found k=0,1,3, but no mathematical reasoning was seen.
- CamperBob2 1y agoSee how this compares to what you got from o3: https://chatgpt.com/share/687bf8bf-c1b0-800b-b316-ca7dd9b00941 https://chatgpt.com/share/687bf8bf-c1b0-800b-b316-ca7dd9b009... It convincingly argues that Gemini's answer was wrong, and Gemini agrees ( https://g.co/gemini/share/aa26fb1a4344 https://g.co/gemini/share/aa26fb1a4344 ). So that's pretty cool, IMO. Pitting these two models against each other in a cage match is an underused hack in my experience. Another observation worth making is that (looking at the Github link) OpenAI didn't just paste an image of the question into the prompt, hit the button and walk away, like I did. They rewrote the prompts carefully to get the best results, and I'm a little surprised people aren't crying foul about that. So I'm pretty impressed with o3-pro's unassisted performance.
- esjeon 1y agoI get the feeling that modern computer systems are so powerful that they can solve almost all well-explored closed problems with a properly tuned model. The problem lies in efficiency, reliability, and cost. Increasing efficiency and reliability would require an exponential increase in cost. QC might solve that cost part, and symbolic reasoning model will significantly boost both efficiency and reliability.
- gitfan86 1y agoThis is such an interesting time because the percentage of people who are making predictions about AGI happening on the future are going to drop off and the number of people completely ignoring the term AGI will increase.
- deleted 1y ago[deleted]
- SiempreViernes 1y agoThat doesn't seem likely because the LLMs haven't really delivered any great products that can cover the money spent and so AGI hype is essentially to keep the money flowing.
- ctoth 1y agoPre-registering a prediction: When (not if) AI does make a major scientific discovery, we'll hear "well it's not really thinking, it just processed all human knowledge and found patterns we missed - that's basically cheating!"
- ysavir 1y agoLess that AI is cheating and more that we basically found a way to take the thousand monkeys with infinite time scenario and condense that into a reasonable(?) amount of time and with some decent starting instructions. The AI wouldn't have done any of the heavy lifting of the discovery, it just iterated on the work of past researchers at speeds beyond human.
- colinplamondon 1y agoHonest question - how is that not true of those past researchers? IE, they... - Start with the context window of prior researchers. - Set a goal or research direction. - Engage in chain of thought with occasional reality-testing. - Generate an output artifact, reviewable by those with appropriate expertise, to allow consensus reality to accept or reject their work.
- peripitea 1y agoIt sounds like you're saying AI is just doing brute force with a lot of force, but I can't imagine that's actually what you think, so would you mind clarifying?
- j_timberlake 1y ago
- famouswaffles 1y agoFrom Noam Brown https://x.com/polynoamial/status/1946478258968531288 https://x.com/polynoamial/status/1946478258968531288 "When you work at a frontier lab, you usually know where frontier capabilities are months before anyone else. But this result is brand new, using recently developed techniques. It was a surprise even to many researchers at OpenAI. Today, everyone gets to see where the frontier is." and "This was a small team effort led by @alexwei_ . He took a research idea few believed in and used it to achieve a result fewer thought possible. This also wouldn’t be possible without years of research+engineering from many at @OpenAI and the wider AI community."
- deleted 1y ago[deleted]
- yahoozoo 1y agoThat brand new technique? Training on the test data. /s
- otabdeveloper4 1y agoHey, cheating is also an artform.
- dinfinity 1y agoDo you have any proof to support this claim?
- gsf_emergency_2 1y ago"frontier" seems to be "zapad" for OpenAI They have a parallel effort to corner Ramanujan called https://epoch.ai/frontiermath/tier-4 https://epoch.ai/frontiermath/tier-4 (& Problem 6, combinatorics, the one class of problems not yet fallen to AI?) The hope for humanity is that of the big names associated to FrontierMath (starkly opposite to oAI proper) Daniel is the one youngish nonexsoviet guy :)
- robinhouston 1y agoThere is some relevant context from Terence Tao on Mathstodon: > It is tempting to view the capability of current AI technology as a singular quantity: either a given task X is within the ability of current tools, or it is not. However, there is in fact a very wide spread in capability (several orders of magnitude) depending on what resources and assistance gives the tool, and how one reports their results. > One can illustrate this with a human metaphor. I will use the recently concluded International Mathematical Olympiad (IMO) as an example. Here, the format is that each country fields a team of six human contestants (high school students), led by a team leader (often a professional mathematician). Over the course of two days, each contestant is given four and a half hours on each day to solve three difficult mathematical problems, given only pen and paper. No communication between contestants (or with the team leader) during this period is permitted, although the contestants can ask the invigilators for clarification on the wording of the problems. The team leader advocates for the students in front of the IMO jury during the grading process, but is not involved in the IMO examination directly. > The IMO is widely regarded as a highly selective measure of mathematical achievement for a high school student to be able to score well enough to receive a medal, particularly a gold medal or a perfect score; this year the threshold for the gold was 35/42, which corresponds to answering five of the six questions perfectly. Even answering one question perfectly merits an "honorable mention". > But consider what happens to the difficulty level of the Olympiad if we alter the format in various ways: > * One gives the students several days to complete each question, rather than four and half hours for three questions. (To stretch the metaphor somewhat, consider a sci-fi scenario in the student is still only given four and a half hours, but the team leader places the students in some sort of expensive and energy-intensive time acceleration machine in which months or even years of time pass for the students during this period.) > * Before the exam starts, the team leader rewrites the questions in a format that the students find easier to work with. > * The team leader gives the students unlimited access to calculators, computer algebra packages, textbooks, or the ability to search the internet. > * The team leader has the six student team work on the same problem simultaneously, communicating with each other on their partial progress and reported dead ends. > * The team leader gives the students prompts in the direction of favorable approaches, and intervenes if one of the students is spending too much time on a direction that they know to be unlikely to succeed. > * Each of the six students on the team submit solutions, but the team leader selects only the "best" solution to submit to the competition, discarding the rest. > * If none of the students on the team obtains a satisfactory solution, the team leader does not submit any solution at all, and silently withdraws from the competition without their participation ever being noted. > In each of these formats, the submitted solutions are still technically generated by the high school contestants, rather than the team leader. However, the reported success rate of the students on the competition can be dramatically affected by such changes of format; a student or team of students who might not even reach bronze medal performance if taking the competition under standard test conditions might instead reach gold medal performance under some of the modified formats indicated above. > So, in the absence of a controlled test methodology that was not self-selected by the competing teams, one should be wary of making apples-to-apples comparisons between the performance of various AI models on competitions such as the IMO, or between such models and the human contestants. Source: https://mathstodon.xyz/@tao/114881418225852441 https://mathstodon.xyz/@tao/114881418225852441 https://mathstodon.xyz/@tao/114881419368778558 https://mathstodon.xyz/@tao/114881419368778558 https://mathstodon.xyz/@tao/114881420636881657 https://mathstodon.xyz/@tao/114881420636881657
- eboynyc32 1y agoThe world is changing and it’s exciting. Either you’re on or you’re off. The world doesn’t wait.
- suddenlybananas 1y agoI don't believe this.
- amelius 1y agoIf someone told me this say, 10 or 20 years ago, I would have assumed this was worthy of a Nobel/Turing prize ...
- raincole 1y agoEarly machine learning researchers literally got Nobel Prize last year. Clearly not every incremental step of progress merits a Nobel.
- amelius 1y agoYes! But there is also a delay as 10 or 20 years ago we already had neural nets and I'm curious if people back then thought the concept was Nobel worthy.
- JohnKemeny 1y agoWe have been doing automated theorem proving since 1954.
- nextworddev 1y agoYep, we are about to get fast takeoff
- hislaziness 1y agoTerence Tao on the matter - https://imgur.com/a/terence-tao-on-supposed-gold-imo-sMKP0bm https://imgur.com/a/terence-tao-on-supposed-gold-imo-sMKP0bm
- mananaysiempre 1y agoActual post instead of ad-decorated screnshot: https://mathstodon.xyz/@tao/114881418225852441 https://mathstodon.xyz/@tao/114881418225852441 (thread continued in https://mathstodon.xyz/@tao/114881419368778558 https://mathstodon.xyz/@tao/114881419368778558 and https://mathstodon.xyz/@tao/114881420636881657 https://mathstodon.xyz/@tao/114881420636881657).
- jiggawatts 1y agoFair points, but the reason everyone is amazed is that five years ago this was entirely impossible for computers irrespective of the competition format or rules. It’s as-if we had learned whale song, and then within two years a whale had won a Nobel prize for their research in high pressure aquatic environments. You’d similarly get naysayers debating the finer points of what special advantage whales may have in that particular field, neglecting the stunned shock of the general population — “Whales are publishing research papers now!? Award winning papers at that!?”
- vineyardmike 1y agoAnd it’s very impressive that whales can write papers. A computer system that can perform these tasks that were unthinkably complex a few years ago is quite impressive. That is a big win, and it can be celebrated. They don’t need to be celebrated as a “gold medalist” if they didn’t perform according to the same criteria as a gold-medalist.
- deleted 1y ago[deleted]
- jiggawatts 1y ago
- ghm2180 1y agoWhile this is nice for splashy headlines, I like the headlines which would read some real life usecase of math grads using AI as a companion tool for solving novel problems.
- jacquesm 1y agoThere's that pesky Fermi Paradox. Who knows, we might solve that one too!
- deleted 1y ago[deleted]
- fxj 1y agojust tried this: take the graph of the functions x^n and exp(x) how many points of intersection do they have? chatgpt gave me the wrong answer, it claimed 2 points of intersection, but for n=4 there are 3 as one can easily derive. one for negative x and 2 points for positive x because exp(x) is growing faster than x^4. then i corrected it and said 3 points of intersection. it said yes and gaev me the 3 points. then i said no there are 4 points of intersection and it again explained to me that there are 2 points of intersection. which is wrong. then i asked it how many points of intersection for n=e and it said: zero well, exp(x)=x^e for x=e, isnt it?
- TestTime_9000 1y ago"The LLM system's core mechanism is probably a "propose-verify" loop that operates on a vocabulary of special tokens representing formal logic expressions. At inference time, it first proposes a new logical step by generating a sequence of these tokens into its context window, which serves as a computational workspace. It then performs a subsequent computational pass to verify if this new expression is a sound deduction from the preceding steps. This iterative cycle, learned from a vast corpus of synthetic proof traces, allows the model to construct a complete, validated formal argument. This process results in a system with abstract reasoning capabilities and functional soundness across domains that depend on reasoning, achieved at the cost of computation required for its extended inference time."