12 ms·
OpenAI researcher announced GPT-5 math breakthrough that never happened
- amelius 1y ago> Summary (from the article) * OpenAI researchers claimed or suggested that GPT-5 had solved unsolved math problems, but in reality, the model only found known results that were unfamiliar to the operator of erdosproblems.com. * Mathematician Thomas Bloom and Deepmind CEO Demis Hassabis criticized the announcement as misleading, leading the researchers to retract or amend their original claims. * According to mathematician Terence Tao, AI models like GPT-5 are currently most helpful for speeding up basic research tasks such as literature review, rather than independently solving complex mathematical problems.
- deleted 1y ago[deleted]
- HarHarVeryFunny 1y ago> GPT-5 had only surfaced existing research that Bloom had missed So GPT-5 didn't derive anything itself - it was just an effective search engine for prior research, which is useful, but not any sort of breakthough whatsoever.
- jgalt212 1y agoAfter the circular financing schemes involving hundreds of billions of dollars were uncovered, nothing I read about the AI business and its artificial hype machine surprises me anymore.
- anskxbsb 1y ago[dead]
- bbor 1y agoThis is just tit-for-tat clickbait. The researcher’s wording was a bit unclear for sure, but far from incorrect.
- resoluteteeth 1y agoI disagree. There is no way to interpret "GPT-5 just found solutions to 10 (!) previously unsolved Erdos problems" as saying something other than GPT-5 having solved them. If it just found existing solutions then they obviously weren't "previously unsolved" so the tweet is wrong. He clearly misunderstood the situation and jumped to the conclusion that GPT-5 had actually solved the problems because that's what he wanted to believe. That said, the misunderstanding is understandable because the tweet he was responding to said they had been listed as "open", but solving unsolved erdos problems by itself would be such a big deal that he probably should have double checked it.
- deleted 1y ago[deleted]
- bbor 1y agoBut… it found them… and they were previously marked unsolved… Note that “solved” does not equal “found”
- deleted 1y ago[deleted]
- andrewstuart 1y agoHumans hallucinating about AI.
- deleted 1y ago[deleted]
- MattGaiser 1y agoHumans "hallucinate" in the AI way constantly, which is why I don't see them as a barrier to LLMs replacing humans in many contexts. It really isn't unusual for a human to make stuff up or be unaware of stuff.
- pas 1y agoit's the same thing with self-driving, if you can make it safer than a good human driver it's enough. but the bar is pretty low with driving (as evidenced by the hundreds of thousands of collisions and deaths and permanent disabilities each year). and rather high in scientific publishing.
- zeknife 1y agoA human being informed of a mistake will usually be able to resolve it and learn something in the process, whereas an LLM is more likely to spiral into nonsense
- alimw 1y agoYou must know better humans than I do.
- MattGaiser 1y agoYou must know people without egos. Humans are better at correcting their mistakes, but far worse at admitting them. But yes, as an edge case handler humans still have an edge.
- topaz0 1y agoLLMs by contrast love to admit their mistakes and self-flagellate, and then go on to not correct them. Seems like a worse tradeoff.
- mentalgear 1y agoAnother instance of openAI manipulating results to prolong their unsustainable circular hype bubble. The inevitable collapse could be even more devastating than the 2008 financial crisis. All while so vast resources are being wasted on non-verifiable gen AI slob, while real approaches (neuro-symbolic like DeepMind's AlphaFold) are mostly ignored financially because they don't generate the quick stock market increases that hype does.
- the_duke 1y agoPeople keep spouting this, but I don't see how the AI bubble bursting would be all that devastating. 2008 was a systemic breakdown rippling through the foundations of the financial system. It would lead to a market crash (80% of gains this year were big tech/AI) and likely a full recession in the US, but nothing nearly as dramatic as a global systemic crisis. In contrast to the dot com bubble, the huge AI spending is also concentrated on relatively few companies, many with deep pockets from other revenue sources (Google, Meta, Microsoft, Oracle), and the others are mostly private companies that won't have massive impact on the stock market. A sudden stop in AI craze would be hard for hardware companies and a few big AI only startups , but the financial fallout would be much more contained than either dot com or 2008.
- Topfi 1y agoIsn't the dot com bubble a far better proxy? Notably, todays spending is both higher and more concentrated in a few companies that a large part of the population has exposure to (most dot com companies weren't publicly traded and far smaller vs MSFT, Alphabet, Meta, Oracle, NVDA making up most investment today) by way of pension funds, ETFs, etc.?
- the_duke 1y agoSure, but all of the above have solid businesses that rake in lots of money, revenue based on AI is a small percentage for them. An AI bust would take the stock price down a good deal, but the stock gains have been relatively moderate. Year on year: Microsoft +14%, Meta +24%, Google +40, Oracle +60%, ... And a notable chunk of those gains have indirectly come from the dollar devaluing. Nvidia would be hit much harder of course. There is a good amount of smaller AI startups, but a lot of the AI development is concentrated on the big dogs, it's not nearly as systemic as in dot com, where a lot of businesses went under completely. And even with an AI freeze, there is plenty of value and usage there already that will not go away, but will keep expanding (AI chat, AI coding, etc) which will mitigate things.
- kif 1y agoThis honestly doesn’t surprise me. We have reached a point where it’s becoming clearer and clearer that AGI is nowhere to be seen, whereas advances in LLM ability to ‘reason’ have slowed down to (almost?) a halt.
- dawnerd 1y agoBut if you ask an AI hype person they’ll say we’re almost there we just need a bit more gigawatts of compute!
- rhetocj23 1y agoI hate to say this but I think the LLM story is going to go the same way as Teslas stock - everyone knows its completely detached from fundamentals and driven by momentum and hype but nobody wants to do the right thing.
- vbezhenar 1y agoIn my book, chat-based AGI has been reached years ago, when I couldn't reliably distinguish computer from human. Solving problems that humanity couldn't solve is super-AGI or something like that. It's not there indeed.
- 3836293648 1y agoBeating the Turing Test is not AGI, but it is beating the Turing Test and that was impressive enough when it happened
- mrguyorama 1y agoSo you were impressed by ELIZA right? Because that's what first "beat the turing test" Which, actually is not a real thing. Nor has it ever really been meaningful. Trolls on IRC "beat the turing test" with bots that barely even had any functionality.
- 1y ago
- Analemma_ 1y ago“AGI achieved internally” Another case of culture flowing from the top I guess.
- strangescript 1y agoThis entire thing has been pretty disingenuous on both sides of the fence. All the anti-AI (or anti OpenAI) people are doing victory laps, but what GPT-5 Pro did is still very valuable. 1) What good is your open problem set if really its a trivial "google search" away from being solved. Why are they not catching any blame here? 2) These answers still weren't perfectly laid out for the most part. GPT-5 was still doing some cognitive lifting to piece it together. If a human would have done this by hand it would have made news and instead the narrative would have been inverted to ask serious questions about the validity of some these style problem sets and/or ask the question how many other solutions are out there that just need pieced together from pre-existing research. But, you know, AI Bad.
- nurettin 1y agoAI great, but AI not creative, yet.
- puttycat 1y agoThis is a strawman argument. No anti-AI sentiment was involved here. Simply the fact that finding and matching text on the Internet is several orders of magnitude easier than finding novel solutions to hard math problems.
- strangescript 1y agoYou didn't read the X replies if you believe that
- matsemann 1y agoYou're moving the goal post.
- Topfi 1y ago> What good is your open problem set if really its a trivial "google search" away from being solved. Why are they not catching any blame here? They are a community run database, not the sole arbiter and source of this information. We learned the most basic research back in highschool, I'd hope researchers from top institutions now working for one of the biggest frontier labs can do the same prior to making a claim, but microblogging has and continues to be a blight on any accurate information so nothing new there. > GPT-5 was still doing some cognitive lifting to piece it together. Cognitive lifting? It's a model, not a person, but besides that fact, this was already published literature. Handy that a LLM can be a slightly better search, but calling claims of "solving maths problems" out as irresponsible and inaccurate is the only right choice in this case. > If a human would have done this by hand it would have made news [...] "Researcher does basic literature review" isn't news in this or any other scenario. If we did a press release every journal club, there wouldn't be enough time to print a single page advert. > [...] how many other solutions are out there that just need pieced together from pre-existing research [...] I am not certain you actually looked into the model output or why this was such an embarrassment. > But, you know, AI Bad. AI hype very bad. AI anthropomorphism even worse.
- deleted 1y ago[deleted]
- random9749832 1y agoYou are telling me a language model trained on Reddit can't solve novel problems? Shocking. Edit: we are in peak damage control phase of the hype cycle.
- creativeCak3 1y agoCan't wait for the Buble to burst so we can get back to solving real problems (like the fact that we have very little competition in the CPU market right now, AMD is getting way too comfortable...). I do think though that when this bubble bursts it will hurt the machine learning field (which does have people doing practically useful stuff like protein folding) and investors might pull all funding because this generative nonsense(which has no real use beyond generating porn) will taint the entire field, even the parts of it that are actually useful.
- phplovesong 1y agoHow fing obvious was it that AI slop did not do anything other than scarpe some websites.
- Jweb_Guru 1y agoI felt like I was going crazy when people uncritically accepted the original claim from OpenAI. Have people actually used these models?
- phplovesong 1y agoFrom what i have seen, people using AI somehow get the mindset that the AI generated result is "godlike" and the "ultimate truth". Its really, really scary and im not that hopeful for what we will see int he next decade. Once i told a coworker that a piece if his code looked rather funky (without doing a more deep CR), and he told me its "proven correct by AI". I was stunned, and asked him if he knows how LLMs generate their responses? He was genuinely in the belief that it was in fact "artificial intelligence" and was some sort of "all knowing entity".
- Jweb_Guru 1y agoIt just drives me nuts when I see people say things like "yeah I asked ChatGPT about this extremely famous open problem, wish me luck!" Like what do you expect to happen exactly with an engine that can't even consistently keep track of what you wrote ten thousand tokens ago?
- deleted 1y ago[deleted]
- amirhirsch 1y agoThe sad truth about this incident is that it reveals that OpenAI does not have a serious effort to actually work on unsolved math problems.
- rowanG077 1y agoHow so? I wouldn't put much stock into a roque employee announcing something wrong.
- mrbungie 1y agoThat's not any employee, its their VP of Science.
- amirhirsch 1y agoThe people involved are very smart and must know that AI doing novel math is a canary for AGI. A serious effort around solving open problems would not fuck up this kind of announcement.
- coldtea 1y ago"rogue employee"
- jebarker 1y agoThat’s a non sequitur. They’re a fairly large organization, I’d be amazed if they don’t have multiple research sub-teams pursuing all sorts of different directions.
- amirhirsch 1y agoIt is not about whether some team somewhere is dabbling in math, it’s about institutional emphasis. When serious math work exists, it tends to be checked before being broadcast. The fact that this didn’t happen suggests that genuine mathematical rigor isn’t a central focus.
- grafmax 1y ago
- llm_nerd 1y agoYann LeCun's "Hoisted by their own GPTards" is fantastic.
- frays 1y agoI might be missing context here, but I'm surprised to see Yann using language that plays on 'retard.' That seems out of character for him - more like something I'd expect from Elon Musk. What's the context I'm missing?
- deleted 1y ago[deleted]
- nova22033 1y agoIt's a play on the word petard
- microtonal 1y agoI found this background useful as a non-native speaker: https://en.wikipedia.org/wiki/Hoist_with_his_own_petard https://en.wikipedia.org/wiki/Hoist_with_his_own_petard
- grey-area 1y agoHoist (thrown in the air) by your own petard (bomb) is a common phrase.
- znkr 1y agoI don’t think it’s a wordplay with the r-word, but rather a reference to the famous Shakespeare quote: “Hoist with his own petard”. It’s become an English proverb. (A petard is a smallish bomb)
- card_zero 1y agoFrom péter, to fart. Possibly entered the language as a saying due to Shakespeare being scurrilous.
- throwawayerdos 1y ago[dead]
- JKCalhoun 1y agoI try not to lose sight of the first time that I heard (some years back) that people were using this new LLM thing for DM'ing ("dungeon mastering", leading) a game of Dungeons and Dragons. I thought, this must be bullshit or some kind of witchcraft. Definitely not anti-AI here. I think I have been disappointed though, since then, to slowly learn that they're (still) little beyond that. Still amazing though. And better than a Google search (IMHO).
- Palmik 1y agoThe original tweet was clearly misunderstood... https://x.com/SebastienBubeck/status/1977181716457701775 https://x.com/SebastienBubeck/status/1977181716457701775: > gpt5-pro is superhuman at literature search: > it just solved Erdos Problem #339 (listed as open in the official database https://erdosproblems.com/forum/thread/339 https://erdosproblems.com/forum/thread/339) by realizing that it had actually been solved 20 years ago https://x.com/MarkSellke/status/1979226538059931886 https://x.com/MarkSellke/status/1979226538059931886: > Update: Mehtaab and I pushed further on this. Using thousands of GPT5 queries, we found solutions to 10 Erdős problems that were listed as open: 223, 339, 494, 515, 621, 822, 883 (part 2/2), 903, 1043, 1079. It's clearly talking about finding existing solutions to "open" problems. The main mistake is by Kevin Weil, OpenAI CTO, who misunderstood the tweet: https://x.com/kevinweil/status/1979270343941591525 https://x.com/kevinweil/status/1979270343941591525: > you are totally right—I actually misunderstood @MarkSellke's original post, embarrassingly enough. Still very cool, but not the right words. Will delete this since I can't edit it any longer I think. Obviously embarassing, but completely overblown reaction. Just another way for people to dunk on OpenAI :)
- Topfi 1y agoIf holding the CTO of OpenAI accountable for his wildly inaccurate statement constitutes "dunking on OpenAI", then I'd say dunk away. He, more than anyone else, should be able to for one parse the original statements correctly and for another maybe realize that if one of their models had accomplished what he seemed to think GPT-5 had, that may require some more scrutiny and research before posting it. That would have, after all, been a clear and incredibly massive development for the space, something the CTO of OpenAI should recognize instantly. The amount of people that told me this is clear and indisputable proof that AGI/ASI/whatever is either around the corner or already here is far more than zero and arguing against their misunderstanding was made all the more challenging because "the CTO of OpenAI knows more than you" is quite a solid appeal to authority. I'd recommend maybe a waiting period of 48h before any authority in any field can send a tweet, that might resolve some of the inaccuracies and the incredibly annoying need to just jump on wild bandwagons...
- zozbot234 1y ago"you are totally right—I actually misunderstood" ...like, seriously? Did an AI come up with this retraction, or are humans actually talking like robots now?
- 827a 1y agoThis happening the same week as DeepMind’s seemingly legitimate AI-assisted cancer treatment breakthrough is a startlingly bad look for OpenAI. My boss always used to say “our only policy is, don’t be the reason we need to create a new policy”. I suspect OpenAI is going to have some new public communication policies going forward.
- gpjt 1y agoTo be fair to the OpenAI team, if read in context the situation is at worst ambiguous. The deleted tweet that the article is about said "GPT-5 just found solutions to 10 (!) previously unsolved Erdös problems, and made progress on 11 others. These have all been open for decades." If it had been posted stand-alone then I would certainly agree that it was misleading, but it was not. It was a quote-tweet of this: https://x.com/MarkSellke/status/1979226538059931886?t=OigN6tLzp8ZYRUCthf0OCg&s=09 https://x.com/MarkSellke/status/1979226538059931886?t=OigN6t..., where the author is saying he's "pushing further on this". The "this" in question is what this second tweet is in turn quote-tweeting: https://x.com/SebastienBubeck/status/1977181716457701775?t=TnV_l7QQzHONs9QVs5So3w&s=19 https://x.com/SebastienBubeck/status/1977181716457701775?t=T... -- where the author says "gpt5-pro is superhuman at literature search: [...] it just solved Erdos Problem #339 (listed as open in the official database erdosproblems.com/forum/thread/3…) by realizing that it had actually been solved 20 years ago" So, reading the thread in order, you get * SebastienBubeck: "GPT-5 is really good at literature search, it 'solved' an apparently-open problem by finding an existing solution" * MarkSellke: "Now it's done ten more" * kevinweil: "Look at this cool stuff we've done!" I think the problem here is the way quote-tweets work -- you only see the quoted post and not anything that it in turn is quoting. Kevin Weil had the two previous quotes in his context when he did his post and didn't consider the fact that readers would only see the first level, so wouldn't have Sebastien Bubek's post in mind when they read his. That seems like an easy mistake to entirely honestly make, and I think the pile-on is a little unfair.
- deleted 1y ago[deleted]
- card_zero 1y agoSo the first guy said "solved [...] by realizing that it had actually been solved 20 years ago", and the second guy said "found solutions to 10 (!) previously unsolved Erdös problems". Previously unsolved. The context doesn't make that true, does it?
- glenstein 1y agoRight, and I would even go a step further and say the context from SebastienBubeck is stretching "solved" past its breaking point by equating literature research with self-bootsrapped problem solving. When it's later characterized as "previously unsolved" it's doubling down on the same equivocation. Don't get me wrong, effectively surfacing unappreciated research is great and extremely valuable. So there's a real thing here but with the wrong headline attached to it.
- ripped_britches 1y agoI make mistakes all the time. This seems like a genuine mistake, not malice. Imagine if you were talking about your own work online, you make an honest mistake, then the whole industry roasts you for it. I’m so tired of hearing everyone take stabs at people at OpenAI just because they don’t personally like sama or something.
- strongbond 1y agoMaybe Open AI shouldn't be so stabable?
- manquer 1y agoMistakes are human yes we all do make them all time. However when representing an reputable organization, people are expected to be cautious or otherwise required to have their comments reviewed and most organizations would enforce this to protect their brand or reputation. As Carl Sagan said it best, extraordinary claims require extraordinary evidence. This was a pretty extraordinary claim and multiple senior staff at the org endorsed the comment without even a cursory check first. I would think serious observers are more concerned by the process and controls in OpenAI or lack thereof here, rather than a specific single mistake. Like them or hate them, OpenAI is the leader in the industry, and everyone lookup up to them and their employees in a public forum for credible information so will hold them to higher standard than a lesser known lab. The burden of checking for quality comes with having a reputable brand . The burden is being monetized / compensated with the valuation or size of fund raise a org is able to command.
- Timsky 1y ago> GPT-5 is proving useful as a literature review assistant No, it does not. It only produces a highly convincing counterfeit. I am honestly happy for people who are satisfied with its output: life is way easier for them than for me. Obviously, the machine discriminates me personally. When I spend hours in the library looking for some engineering-related math made in the 70s-80s, as a last resort measure, I can try to play this gambling with chat, hoping for any tiny clue to answer my question. And then for the following hours, I am trying to understand what is wrong with the chat output. Most often, I experience the "it simply can't be" feeling, and I know I am not the only one having it.
- glenstein 1y agoStruggling to understand this one. Is it that (1) it's lopsided toward reference materials found on the modern internet and not as useful for reviewing literature from the Before Times or (2) it's offering specific solutions but you're skeptical of them?
- crazygringo 1y agoIn my experience doing literature super-deep-dives, it hallucinates sources about 50% of the time. (For higher-level literature surveys, it's maybe 5%.) Of the other 50% that are real, it's often ~evenly split into sources I'm familiar with and sources I'm not. So it's hugely useful in surfacing papers that I may very well never have found otherwise using e.g. Google Scholar. It's particularly useful in finding relevant work in parallel subfields -- e.g. if you work in physics but it turns out their are math results, or you work in political science and it turns out there are relevant findings from anthropology. And also just obscure stuff -- a random thesis that never got published or cited but the PDF is online and turns out to be relevant. It doesn't matter if 75% of the results are not useful to me or hallucinated. Those only waste me minutes. The other 25% more than make up for it -- they're things I simply might never find otherwise.
- macrolime 1y agoWhat is "it". Gpt-5 auto? Gpt-5 pro? Deep research? These have wildly different hallucination rates.
- cedws 1y agoMaking such a claim should at the very least require proof that the information wasn’t in the training data.
- d_tr 1y agoWhatever happened to "don't get high on your own supply"?
- DonHopkins 1y agoMan, that's soap, man!!! https://www.youtube.com/watch?v=RvGE-xhroy0 https://www.youtube.com/watch?v=RvGE-xhroy0 [drinks pee twice]
- deleted 1y ago[deleted]
- YesBox 1y agoWouldn't be surprised if OpenAI employees are being asked to phrase (market) things this way. This is not the first time they claimed GPT-5 "solved" something [1] [1] https://x.com/SebastienBubeck/status/1970875019803910478 https://x.com/SebastienBubeck/status/1970875019803910478 edit: full text It's becoming increasingly clear that gpt5 can solve MINOR open math problems, those that would require a day/few days of a good PhD student. Ofc it's not a 100% guarantee, eg below gpt5 solves 3/5 optimization conjectures. Imo full impact of this has yet to be internalized...
- deleted 1y ago[deleted]
- deleted 1y ago[deleted]
- flkiwi 1y agoThanks for calling that out. You're right to be upset.
- sbaidon94 1y agoYou would think Open AI employees have a pretty good grasp of their model capabilities, but even if you don’t, you probably always want to be on the cautious side for every claim you see on the internet. This just seems to be the Open AI culture, which for better or worse has helped foster the AI hype environment we are currently in.
- bwfan123 1y ago"It is difficult to get a man to understand something, when his salary depends upon his not understanding it,"
- ath3nd 1y ago[dead]
- kenjackson 1y ago“Mathematician Thomas Bloom, who runs erdosproblems.com, pushed back right away. He called the statements "a dramatic misinterpretation," clarifying that "open" on his site just means he personally doesn't know the solution - not that the problem is actually unsolved.” What mathematician uses this as the definition for “open”? I don’t go around saying that most problems in this textbook are open questions, just because I don’t know how to do them.
- ModernMech 1y agoHe's pointing out that erdosproblems.com is an eventually consistent list of solved problems, so there may be a lag between when a problem is solved and when it's marked as such on his site.
- imtringued 1y agoIf a book lists an open problem and you solve it, how would the book know?
- kenjackson 1y agoDo people here really not know what an open problem in math is?
- ares623 1y agoImagine having your retirement fund manager investing in a couple million more based on the original tweet. Very cool.
- classified 1y agoWas that "researcher" just a hallucinating LLM?