6 ms·

An internal OpenAI Astra model solved 10 major open math and CS problems

49 points by wa5ina 2mo ago

HarHarVeryFunny 2mo ago
I feel like this kind of "result dump" just cheapens mathematics. How about having a little respect for those whose work this builds on, and current mathematicians some of who may have spent years working on these problems. Rather than sitting on these results until they had enough for a "shock and awe" 10-result dump, how about releasing these results individually as they were made/verified, as well as the failures (equally valuable to assess the current capabilities of LLMs), and try to make some analysis of HOW these breakthrough results were made. What were the prompts for each of these, how much guidance was there from the mathematicians employed by OpenAI, and most importantly how did the model arrive at these results ... what lines of reasoning resulted it in exploring ideas that humans had previously not explored?
- shshshsbe 2mo ago
  I know it’s tough, but I am not a fan of elitism. Mathematics is no different from all other branches which themselves are just intellectual labor which is again just a special type of labor. There is nothing magical about it and if computers can trivialize it, so be it. Where were all the mathematicians and academics in general when “regular joe” was automated? Now it’s hitting close to home and their foreheads are starting to get sweaty. I’d say let them. Tough luck. Make mathematics as “cheap” as possible. Nobody owes them any favors. Let’s commoditize “being smart” and let go of arbitrary divisions between us.
  - false-mirror 2mo ago
    I'd prefer the reverse approach. Let's value the "regular Joe" instead of cheapening everyone.
    - znjssjnsns 2mo ago
      Yes, but in practice that doesn’t happen. Again, I did not hear a loud protest against automation of all types of labor from the intelligentsia in the past. In practice automation is great, as long as it doesn’t hit “the ones that matter” (a label which they themselves assign). I find it very hard to not imagine the smallest, tiniest violin playing the saddest song for them. Again, “respect for mathematicians” and “their work”.. please. Just produce results. That’s all that ever mattered and let’s not change the rules of the game just because they don’t suit you anymore.
      - skew-aberration 2mo ago
        Protests against automation, or for the restructuring of society to accommodate automation without causing suffering of laborers, etc, is literally one of the largest and most extensively discussed topics in modern history.
    - zer00eyz 2mo ago
      This is what the Ludites were trying to accomplish with an (armed) uprising. Go back 100 years and 15 percent of the population was "Farmer", and 30 percent were in some sort of "domestic" work. Most modern jobs are a product of the fact that technology continues to advance. And by all measures it does not look like "ai" is going to change that.
    - rwz 2mo ago
      If everybody is "valued", nobody is. The concept of value implies exclusivity by design.
  - robotpepi 2mo ago
    that's an incredibly narrow view of what's happening.
  - deleted 2mo ago
    [deleted]
- alanbernstein 2mo ago
  Not that I disagree, but I think this is how many people feel about AI output in fields they care about. It's also a funny historical mirror to an earlier phase of math proof culture: in a previous era, cryptic result dumps were quite common.
- seanmcdirmid 2mo ago
  You can actually do all of that meta analysis if you have the conversation that led to the solution. This is as easy as just having people release logs of their conversations, and then you can load it into another LLM as context to ask a bunch of questions about it. I hope this will be normal practice eventually. “Show your work” is trivial if you use AI to solve a problem.
- deleted 2mo ago
  [deleted]
- moezd 2mo ago
  That is the point: Make "X can be done by AI" common discourse and watch industries "being disrupted". At least with open weights we'd be able to verify the claims.
- znjssjnsns 2mo ago
  It is so incredibly liberating to see the intellectual elite struggling with was already reality for 99% of the rest of us: your understanding is not required, it might in fact be detrimental as it amounts to a handbrake on progress. Understanding never was the goal, results were. We will be getting boatloads of those. What does your understanding get us? You get a fancy house out of it and you might be intellectually stimulated by it, sure, but I hope you can see how that does not constitute a valid need for the rest of society to labor just to support your class and its lifestyle.
  - robotpepi 2mo ago
    what are even "results" in mathematics if not understanding? for instance, what's the utility of the counterexamples built by openAI? It seems you're fighting an hallucinated enemy.
    - somenameforme 2mo ago
      One would think something like the study of the square root of negative 1 would have little utility beyond a hand-wavy appeal to understanding, but it turns out to have major practical applications ranging from electrical engineering to 3d rotations. I tend to believe we're still likely at a very primitive level of technology. It's difficult to imagine this primarily because of unknown unknowns. For instance at the end of the 19th century it was believed that science had basically been finished, and all that remained was working things out to ever higher levels of precision to wrap up a few curiosities. Then along come relativity, quantum mechanics, and more - all completely revolutionizing science, and upending centuries of false beliefs. I think we're far more likely to be at the end of the 19th century again, rather than at the end of science, or anywhere near such. And if that's the case, then it may well be that seemingly academic discoveries of today, perhaps ultimately achieved by an LLM, are what help unlock the next great revolution.
      - HarHarVeryFunny 2mo ago
        Yes, but what will truly advance science is new discoveries that require curiosity and creative thinking, and LLMs are not the technology for this. As impressive as these mathematical results are, all they represent is known mathematical techniques being combined in ways that humans had not explored. The AI is not inventing areas of math, or even creating it's own conjectures. There will be more of these to follow, but eventually the pace will slow as the space of what can be done with known mathematics is explored. What is needed for creativity, for new scientific discovery, is more than an auto-regressive LLM assembling someone else's Lego blocks in novel ways - it will require an AI not just built to predict but one that is also built to expand the frontier of what it knows. Prediction failure that results in curiosity and exploration, not hallucination. We'll get there eventually.
- yomismoaqui 2mo ago
  Anti-AI rethoric is getting ridiculous.
  - HarHarVeryFunny 2mo ago
    Go back and re-read. Did I say anything bad about AI? What I'm talking about is the way OpenAI have handled this. They are in the business of artificial intelligence, striving to create human-level, or even super-human intelligence, and telling us what a wonderful future this will create. Shouldn't they be celebrating intelligence, especially human intelligence which is what they have bottled here?! LLMs are themselves minimally intelligent - they are primarily regurgitating human intelligence. Without their human training data, these systems would be nothing. Let's see OpenAI build "GPTZero", an actually intelligent system, that learns for itself, learns language and mathematics from nothing like a baby, then I feel they'd have the right to minimize and cheapen the achievement if they chose to (unlikely). As it is, all they have is a predictor, utterly dependent on HUMAN intelligence, and it would be more becoming of them to acknowledge this by paying a bit more deference and respect to the people who enabled them to build what they have. FWIW I am not a mathematician.
    - yomismoaqui 2mo ago
      They are tools that can help humans do things like disproving the Jacobian Conjecture that very gifted human mathematicians couldn't do until now (even with the help of computers). Yes,their intelligence is jagged, excellent for some things and not so good for others, but calling them "minimally intelligent" is disingenuous.
anon373839 2mo ago
The LLM marketing loop is getting awfully long in the tooth. I saw a meme on Twitter the other day that showed a circular state diagram with something like: > “GPT solved a math problem” -> “Claude solved a math problem” -> “GPT escaped the sandbox” -> “Claude escaped the sandbox” -> …
sunandsurf 2mo ago
Does anyone else have trouble telling how much of this news (along with the 'AI escaping and hacking' stories) is genuine, vs how much is just AI firms overstating their capabilities due to strong commercial incentives?
- Turskarama 2mo ago
  Unlike the hacking one, this would be impossible to bullshit as long as the proofs are released. They can be verified independently, and the alternative is that they solved 10 major open mathematical questions without the AI, which seems less likely.
- sajithdilshan 2mo ago
  I mean once they release the proof you can check the maths yourself and I’m pretty sure it would be peer verified as well.
- watwut 2mo ago
  We hacked a company and blame the tool! Somehow it is not negligence, but cool! We hacked 3 companies and tripple blame the tool! We are even cooler! We hacked companies amd therefore other peoples models need to be restricted! See, us accidentally pointing a hacking tool on others proove we are the only ones that can be trusted with the tool!
- HarHarVeryFunny 2mo ago
  These math results appear genuine, and impressive, and it seems they've already been at least provisionally verified. Of course there is still a massive marketing aspect to this, with the AI companies wanting to you assume that because their product is world-class at math, a capability that is useless to 99.99% of their potential customers, that it will be equally useful in areas that you actually care about, such as managing your vending machine, perhaps :-) It's hard to know how to interpret these hacking confessions/boasts and what the reality is behind them. I get the impression (with low confidence) that they really did not anticipate or orchestrate these attacks, but it also seems they did little to prevent them, and seem to be happy that they occurred (as you note, a chance to suggest how powerful they are). I do think that LLMs/agents can be highly capable, and dangerous as hackers, especially if you deliberately train them to be as was the case with Mythos. There is current news of US water treatment plants being hacked, apparently by Iranian state actors, and we should be glad it was just water treatment plants and not some more critical piece of infrastructure (perhaps power generation or transport, etc). I hate to think what a malicious actor could do with today's SOTA AI if they really wanted to do something destructive, not just send a warning shot.
firesteelrain 2mo ago
I believe this is what we wanted computers to help us solve along with other prior hard problems prior to computers. This should be viewed as a good thing even if Anthropic, OpenAI, etc benefit just like IBM benefitted from mainframes.
- fidotron 2mo ago
  Indeed. I specifically recall my logic lecturers in a compsci course expressing the view the one purpose of computers is to work towards enabling automatic proofs, partly because they saw this as the universal problem in compsci (esp compilers) as well as massive potential for scientific progress generally.
deleted 2mo ago
[deleted]
sajithdilshan 2mo ago
I’m looking forward to the days where AI would help in tackling the problems in biology. Especially, on creating new drugs, enzymes and understanding the genetic diseases. An absolutely interesting time to live.
- croemer 2mo ago
  Right now it makes your account completely useless if you work on something completely harmless related to viruses. Blocks, downgrades, throttling of answers "to check for safety". You need to be at a select few institutions to avoid these safety features, which I'm not. University needs to sign some stuff but I that's outside my control.
seydor 2mo ago
By itself not so interesting announcement. I wish the models were open weights so we could at least do some interesting geometry on the math. It s like a rich man showing off his car collection.
atillavanilla 2mo ago
Ok nice... I am waiting for the "math is dead" argument
sunandcode 2mo ago
Next: LLM model made 10 paperclips.
- Aeolos 2mo ago
  Replace paper clips with data centers and we are already there…
gpm 2mo ago
Henry Yuen's (whose work problem 6 builds on) comments on this are worth reading IMO: https://bsky.app/profile/henryyuen.bsky.social/post/3ms2jpchfjc2t https://bsky.app/profile/henryyuen.bsky.social/post/3ms2jpch...
jsnell 2mo ago
Dupe: https://news.ycombinator.com/item?id=49132058 https://news.ycombinator.com/item?id=49132058

49 points by wa5ina 2mo ago

I feel like this kind of "result dump" just cheapens mathematics. How about having a little respect for those whose work this builds on, and current mathematicians some of who may have spent years working on these problems. Rather than sitting on these results until they had enough for a "shock and awe" 10-result dump, how about releasing these results individually as they were made/verified, as well as the failures (equally valuable to assess the current capabilities of LLMs), and try to make some analysis of HOW these breakthrough results were made. What were the prompts for each of these, how much guidance was there from the mathematicians employed by OpenAI, and most importantly how did the model arrive at these results ... what lines of reasoning resulted it in exploring ideas that humans had previously not explored?

shshshsbe 2mo ago
I know it’s tough, but I am not a fan of elitism. Mathematics is no different from all other branches which themselves are just intellectual labor which is again just a special type of labor. There is nothing magical about it and if computers can trivialize it, so be it. Where were all the mathematicians and academics in general when “regular joe” was automated? Now it’s hitting close to home and their foreheads are starting to get sweaty. I’d say let them. Tough luck. Make mathematics as “cheap” as possible. Nobody owes them any favors. Let’s commoditize “being smart” and let go of arbitrary divisions between us.
- false-mirror 2mo ago
  I'd prefer the reverse approach. Let's value the "regular Joe" instead of cheapening everyone.
  - znjssjnsns 2mo ago
    Yes, but in practice that doesn’t happen. Again, I did not hear a loud protest against automation of all types of labor from the intelligentsia in the past. In practice automation is great, as long as it doesn’t hit “the ones that matter” (a label which they themselves assign). I find it very hard to not imagine the smallest, tiniest violin playing the saddest song for them. Again, “respect for mathematicians” and “their work”.. please. Just produce results. That’s all that ever mattered and let’s not change the rules of the game just because they don’t suit you anymore.
    - skew-aberration 2mo ago
      Protests against automation, or for the restructuring of society to accommodate automation without causing suffering of laborers, etc, is literally one of the largest and most extensively discussed topics in modern history.
  - zer00eyz 2mo ago
    This is what the Ludites were trying to accomplish with an (armed) uprising. Go back 100 years and 15 percent of the population was "Farmer", and 30 percent were in some sort of "domestic" work. Most modern jobs are a product of the fact that technology continues to advance. And by all measures it does not look like "ai" is going to change that.
  - rwz 2mo ago
    If everybody is "valued", nobody is. The concept of value implies exclusivity by design.
- robotpepi 2mo ago
  that's an incredibly narrow view of what's happening.
- deleted 2mo ago
  [deleted]
alanbernstein 2mo ago
Not that I disagree, but I think this is how many people feel about AI output in fields they care about. It's also a funny historical mirror to an earlier phase of math proof culture: in a previous era, cryptic result dumps were quite common.
seanmcdirmid 2mo ago
You can actually do all of that meta analysis if you have the conversation that led to the solution. This is as easy as just having people release logs of their conversations, and then you can load it into another LLM as context to ask a bunch of questions about it. I hope this will be normal practice eventually. “Show your work” is trivial if you use AI to solve a problem.
deleted 2mo ago
[deleted]
moezd 2mo ago
That is the point: Make "X can be done by AI" common discourse and watch industries "being disrupted". At least with open weights we'd be able to verify the claims.
znjssjnsns 2mo ago
It is so incredibly liberating to see the intellectual elite struggling with was already reality for 99% of the rest of us: your understanding is not required, it might in fact be detrimental as it amounts to a handbrake on progress. Understanding never was the goal, results were. We will be getting boatloads of those. What does your understanding get us? You get a fancy house out of it and you might be intellectually stimulated by it, sure, but I hope you can see how that does not constitute a valid need for the rest of society to labor just to support your class and its lifestyle.
- robotpepi 2mo ago
  what are even "results" in mathematics if not understanding? for instance, what's the utility of the counterexamples built by openAI? It seems you're fighting an hallucinated enemy.
  - somenameforme 2mo ago
    One would think something like the study of the square root of negative 1 would have little utility beyond a hand-wavy appeal to understanding, but it turns out to have major practical applications ranging from electrical engineering to 3d rotations. I tend to believe we're still likely at a very primitive level of technology. It's difficult to imagine this primarily because of unknown unknowns. For instance at the end of the 19th century it was believed that science had basically been finished, and all that remained was working things out to ever higher levels of precision to wrap up a few curiosities. Then along come relativity, quantum mechanics, and more - all completely revolutionizing science, and upending centuries of false beliefs. I think we're far more likely to be at the end of the 19th century again, rather than at the end of science, or anywhere near such. And if that's the case, then it may well be that seemingly academic discoveries of today, perhaps ultimately achieved by an LLM, are what help unlock the next great revolution.
    - HarHarVeryFunny 2mo ago
      Yes, but what will truly advance science is new discoveries that require curiosity and creative thinking, and LLMs are not the technology for this. As impressive as these mathematical results are, all they represent is known mathematical techniques being combined in ways that humans had not explored. The AI is not inventing areas of math, or even creating it's own conjectures. There will be more of these to follow, but eventually the pace will slow as the space of what can be done with known mathematics is explored. What is needed for creativity, for new scientific discovery, is more than an auto-regressive LLM assembling someone else's Lego blocks in novel ways - it will require an AI not just built to predict but one that is also built to expand the frontier of what it knows. Prediction failure that results in curiosity and exploration, not hallucination. We'll get there eventually.
yomismoaqui 2mo ago
Anti-AI rethoric is getting ridiculous.
- HarHarVeryFunny 2mo ago
  Go back and re-read. Did I say anything bad about AI? What I'm talking about is the way OpenAI have handled this. They are in the business of artificial intelligence, striving to create human-level, or even super-human intelligence, and telling us what a wonderful future this will create. Shouldn't they be celebrating intelligence, especially human intelligence which is what they have bottled here?! LLMs are themselves minimally intelligent - they are primarily regurgitating human intelligence. Without their human training data, these systems would be nothing. Let's see OpenAI build "GPTZero", an actually intelligent system, that learns for itself, learns language and mathematics from nothing like a baby, then I feel they'd have the right to minimize and cheapen the achievement if they chose to (unlikely). As it is, all they have is a predictor, utterly dependent on HUMAN intelligence, and it would be more becoming of them to acknowledge this by paying a bit more deference and respect to the people who enabled them to build what they have. FWIW I am not a mathematician.
  - yomismoaqui 2mo ago
    They are tools that can help humans do things like disproving the Jacobian Conjecture that very gifted human mathematicians couldn't do until now (even with the help of computers). Yes,their intelligence is jagged, excellent for some things and not so good for others, but calling them "minimally intelligent" is disingenuous.

anon373839 2mo ago

The LLM marketing loop is getting awfully long in the tooth. I saw a meme on Twitter the other day that showed a circular state diagram with something like: > “GPT solved a math problem” -> “Claude solved a math problem” -> “GPT escaped the sandbox” -> “Claude escaped the sandbox” -> …

sunandsurf 2mo ago

Does anyone else have trouble telling how much of this news (along with the 'AI escaping and hacking' stories) is genuine, vs how much is just AI firms overstating their capabilities due to strong commercial incentives?

Turskarama 2mo ago
Unlike the hacking one, this would be impossible to bullshit as long as the proofs are released. They can be verified independently, and the alternative is that they solved 10 major open mathematical questions without the AI, which seems less likely.
sajithdilshan 2mo ago
I mean once they release the proof you can check the maths yourself and I’m pretty sure it would be peer verified as well.
watwut 2mo ago
We hacked a company and blame the tool! Somehow it is not negligence, but cool! We hacked 3 companies and tripple blame the tool! We are even cooler! We hacked companies amd therefore other peoples models need to be restricted! See, us accidentally pointing a hacking tool on others proove we are the only ones that can be trusted with the tool!
HarHarVeryFunny 2mo ago
These math results appear genuine, and impressive, and it seems they've already been at least provisionally verified. Of course there is still a massive marketing aspect to this, with the AI companies wanting to you assume that because their product is world-class at math, a capability that is useless to 99.99% of their potential customers, that it will be equally useful in areas that you actually care about, such as managing your vending machine, perhaps :-) It's hard to know how to interpret these hacking confessions/boasts and what the reality is behind them. I get the impression (with low confidence) that they really did not anticipate or orchestrate these attacks, but it also seems they did little to prevent them, and seem to be happy that they occurred (as you note, a chance to suggest how powerful they are). I do think that LLMs/agents can be highly capable, and dangerous as hackers, especially if you deliberately train them to be as was the case with Mythos. There is current news of US water treatment plants being hacked, apparently by Iranian state actors, and we should be glad it was just water treatment plants and not some more critical piece of infrastructure (perhaps power generation or transport, etc). I hate to think what a malicious actor could do with today's SOTA AI if they really wanted to do something destructive, not just send a warning shot.

firesteelrain 2mo ago

I believe this is what we wanted computers to help us solve along with other prior hard problems prior to computers. This should be viewed as a good thing even if Anthropic, OpenAI, etc benefit just like IBM benefitted from mainframes.

fidotron 2mo ago
Indeed. I specifically recall my logic lecturers in a compsci course expressing the view the one purpose of computers is to work towards enabling automatic proofs, partly because they saw this as the universal problem in compsci (esp compilers) as well as massive potential for scientific progress generally.

deleted 2mo ago

[deleted]

sajithdilshan 2mo ago

I’m looking forward to the days where AI would help in tackling the problems in biology. Especially, on creating new drugs, enzymes and understanding the genetic diseases. An absolutely interesting time to live.

croemer 2mo ago
Right now it makes your account completely useless if you work on something completely harmless related to viruses. Blocks, downgrades, throttling of answers "to check for safety". You need to be at a select few institutions to avoid these safety features, which I'm not. University needs to sign some stuff but I that's outside my control.

seydor 2mo ago

By itself not so interesting announcement. I wish the models were open weights so we could at least do some interesting geometry on the math. It s like a rich man showing off his car collection.

atillavanilla 2mo ago

Ok nice... I am waiting for the "math is dead" argument

sunandcode 2mo ago

Next: LLM model made 10 paperclips.

Aeolos 2mo ago
Replace paper clips with data centers and we are already there…

gpm 2mo ago

Henry Yuen's (whose work problem 6 builds on) comments on this are worth reading IMO: https://bsky.app/profile/henryyuen.bsky.social/post/3ms2jpchfjc2t https://bsky.app/profile/henryyuen.bsky.social/post/3ms2jpch...

jsnell 2mo ago

Dupe: https://news.ycombinator.com/item?id=49132058 https://news.ycombinator.com/item?id=49132058