4 ms·

I've been wondering whether AI really is improving rapidly at open problems or we're being fooled. - OpenAI invites researchers to use their models, in fact gi

I've been wondering whether AI really is improving rapidly at open problems or we're being fooled. - OpenAI invites researchers to use their models, in fact giving at least 100,000 researchers free access[1], but there are also those that pay - Internal OpenAI models are reportedly solving open problems at a surprisingly fast rate[2] - But researchers will typically work on open problems. A researcher who is using Codex to make progress on open problems will be feeding it fresh training data on precisely the problems the internal models are evaluated on. - So while it looks like the new models are suddenly solving lots of open problems, they could be significantly piggybacking on human progress, with models "inspired" by the work of researchers from all around the world? This theory predicts that there'll be many more researchers coming forward just like TFA, as sOpenAI announces more solutions. It doesn't assume all of AI progress is a mirage, just that there's plagiarism. [1]: https://openai.com/index/chatgpt-for-academic-researchers/ https://openai.com/index/chatgpt-for-academic-researchers/ [2]: https://xcancel.com/OpenAI/status/2097374643518640382#m https://xcancel.com/OpenAI/status/2097374643518640382#m

Eddy_Viscosity2 23d ago
> they could be significantly piggybacking on human progress, This is AI in a nutshell, its a plagiarism machine. An abstraction layer between vast amounts of stolen human-generated data that filters out the liabilities and accountability for that original theft. Its an IP laundering system.
- wiei 23d ago
  That’s one perspective. I just view it as a thing that can brute force and produce outputs - that it has no way of ‘knowing’ - but doesn’t need to since it’s just running off of probability. No human can compete in that contest. But no llm can compete in the contest of ‘understanding’ and application in the real world - which is where 99% of the value is. I’m very pro AI long term btw but I’m not blinded.
  - throwawayqqq11 22d ago
    Dont forget the holisitic validators/tools in the process. Probabilistics alone likely will not get you here. These rules are human made and without it, frontier models would not be able to compete, likely.
  - foogazi 22d ago
    But it’s not brute force if it’s looking over everyone’s shoulder Brute force would have been solving Navier-Stokes in 88 hours after plagiarizing all known 20th century math When it needs to snoop live on what the actual mathematicians are working on that’s something else
    - wiei 22d ago
      No its happening whilst the human is working with it. The new inputs provided become part of the brute-force. This is what Scam Altman means by 'self-recursive'. Trust me I've seen it happen to myself. I no longer trust ChatGPT. I can see right through his act. Altman is one devious f8k.
  - AnimalMuppet 22d ago
    AI needs humans to encode ideas in words. It needs those ideas to span the space of possibilities of, say, Navier Stokes. Then AI can be, as you say, a terrifyingly effective way to search that space. But when the building-block ideas are still being formed, I'm not sure that AI is good at forming them.
    - wiei 22d ago
      COrrect and this is how labour displacement happens. There are many actions being performed today that can be nicely packaged. Im already working on such a project.
- robocat 22d ago
  That's such an unquantifiable accusation. Plus it is an unfair standard since so many scientists in the past have been caught unethically using the work of others without attribution (and so many more have been accused). In history we also repeatedly see the phenomenon of multiple discovery or simultaneous invention. If that happens to AI because the topic is pregnant, would you call it "plagiarism" just to disparage AI? https://en.wikipedia.org/wiki/Multiple_discovery https://en.wikipedia.org/wiki/Multiple_discovery
  - Eddy_Viscosity2 22d ago
    Your first example is the apt one here. In this case openAI was, allegedly, pilfering the work of the scientists into the AI. How is it an unfair standard. OpenAI stole the work of others to build the AI. That's not different than scientists stealing from other works as their own, or artists copying others work as their own, etc. It's all plagarism. I'm applying the same standard for everybody. As for multiple discovery, this is a thing, but I don't think the AI did a parallel discovery any more than Ray Kroc made the parallel discovery of the MacDonald brother's speedee service system.
wiei 23d ago
I’d argue the invitation of researchers was incredibly strategic. Sam Altman knows what he’s doing. He will happily screw these folks to one-up his competition.
JeremyNT 22d ago
> I've been wondering whether AI really is improving rapidly at open problems or we're being fooled. I think your suspicions are warranted and your explanation seems plausible. If better training data is the reason here, it would still be a case of the models doing something that is in and of itself super useful! The models really can take that data and distill it into solutions for similar problems faster than humans can. This is great! But there's so much vested interest in the AI companies to be opaque about all this, to hype up their models and avoid giving credit to people whose data made everything possible, that they would never tell us this fact if it were true. I feel like so much of the AI hype cycle is like this. The models develop extremely useful capabilities, but it's hard to understand what they really are through the hype. The lies and obfuscation by their owners who have vested interests in capturing the value they provide makes it impossible to take anything they say at face value.
- YeGoblynQueenne 22d ago
  >> If better training data is the reason here, it would still be a case of the models doing something that is in and of itself super useful! The models really can take that data and distill it into solutions for similar problems faster than humans can. This is great! It's perhaps great in the short term although it's not very clear who it's great for. I'm not sure mathematicians find it all so great, I mean. In the long term, if this contrives to destroy the tradition of human mathematics the whole endeavour is self-defeating. In time, there will be nobody left with the knowledge and skills to produce mathematics to train AI to do mathematics. And then we'll be left with no mathematics at all: we'll have no human mathematicians and no AI that can do mathematics, either.
  - boothby 22d ago
    I was pretty depressed when I read about what happened with Navier Stokes this morning. The Clay Math prizes were a significant motivation through my math career, and I know a lot of computer scientists and physicists that feel similarly. I didn't think I was gonna resolve P vs NP or the BSD conjecture, but I did really research that felt like I was working towards something incredible. What is the younger generation left with? Hey kids betcha can't resolve the Collatz conjecture, our superintelligence can't either! Still pretty depressed about it, to be honest. Intellectualism is dead. We can return to happy agrarianism, I guess. At least the AI doesn't wanna eat my snap peas.
mikgp 22d ago
A mental model I was thinking about was - I remember when Travis Kalanick was talking about using the chatbot to discuss “vibe physics-ing” on the all-in podcast. And like - I think there’s a presumption you could make that AI models could overfit to asymptote towards just the capabilities and knowledge we currently have. And that would be amazing! And crazy useful. And there are probably a whole world of complex problems that remain unsolved because they’re adjacent to knowledge we have but they haven’t been invested in. But can a human reliably tell the difference between “can do 99.999% of the things we currently know how to do which includes a small subset of things we didn’t know we had the capacity to do” and “super intelligent math and science research pushing the frontier of what we know” A physicist that knows all the things we currently know in excruciating detail feels like it should be able to make the leap beyond the frontier. But since these are computer models it might just be that it can ride that line extraordinarily well while the line remains firm.
mannanj 22d ago
It tells me that AI companies are just another mechanism to extract and extort value from the masses for the rich. Just another rich man’s trick Perhaps the last one before they destroy that world and try to hide away as people forget and history is rewritten again. I don’t think they’ll succeed this time.
- dgellow 22d ago
  AI providers are pretty much the end boss of rent seeking, that’s for sure
glitchc 22d ago
The pudding is in the proof. The field is mathematics, the proof can be rigorously verified. If there is a flaw, OpenAI is out to lunch. If the proof is valid, OpenAI has produced something new.
- amelius 22d ago
  Did you read what they said? The question is now if OAI produced something new or just stole the researchers' good ideas.
  - jsLavaGoat 22d ago
    Name one discovery ever that didn't depend on someone else's work.
    - amelius 22d ago
      Most discoveries did not happen by someone looking in someone else's notebooks without them knowing.
  - glitchc 22d ago
    You seem to be unfamiliar about how research works. It's common to make an incremental advancement while citing prior work. The vast majority of papers out there fall into this bucket. Did the AI make incremental progress? Yes. Did it cite prior art? After some nudging, yes. It seems to me the academics are upset that AI scooped them. But scooping is a time-honored tradition between researchers. First to print and all that. In a nutshell, they are upset that they lost out on a publication. I will also point out for those unaware that any mathematics that is produced is automatically part of the public domain and can be used freely in derivative works. It is not a protected intellectual class like other works of art.
    - fg137 22d ago
      > But scooping is a time-honored tradition between researchers. Provided that it's properly accredited. And definitely not for others' unpublished work -- that's despised upon if not an academic integrity issue. People even point out that you should add a reference to certain papers during the peer review process.
bwfan123 22d ago
there are also attempts to crowdsource human research directions - like the caltech mathathon challenge : https://mathathonchallenge.com https://mathathonchallenge.com these would help models on the same problems at the expense of the researchers. basically, math researchers are the reverse centaurs but they dont realize it.
- GPerson 22d ago
  There is a very active open letter of over 1000 signatures from mathematicians in protest of this event. This event is targeting undergraduates. It previously suggested that math researchers already have no place in mathematics, and presents a limited and heavily distorted view of what mathematics research is.
  - andrepd 22d ago
    I'm an AI skeptic, but I don't see how this squares with what the organisers of the event actually say. "It previously suggested that math researchers already have no place in mathematics"? I don't see this.
    - GPerson 22d ago
      The website previously said, “What is the role of a mathematician when AI can solve conjectures faster?” but they have removed it, possibly as a result of the letter since it happened after.
    - GPerson 22d ago
      Also I want to mention that the letter is not about AI skepticism, in any direct way at least.
agumonkey 22d ago
Seems easy to picture high stakes startup cutting corners to justify their fame.
YeGoblynQueenne 22d ago
>> Internal OpenAI models are reportedly solving open problems at a surprisingly fast rate[2] Maybe I'm failing to read that graph properly but the y axis says "pass rate" and it only goes up to 0.5. That would mean every single problem is at most half-solved. I don't know what that means though. What is "0.5 pass rate" in the context of "open math problems" (as in the graph title)?
- red75prime 22d ago
  I guess it's a fraction of problems on which a model produces a LEAN proof or a counterexample.
  - YeGoblynQueenne 22d ago
    Wouldn't they just list the number of problems solved then?
    - dekhn 22d ago
      rates beat counts almost always.
      - YeGoblynQueenne 21d ago
        You're assuming too much from a single graph with vaguely named axes. I'll wait until there's more concrete information.
boothby 22d ago
> - OpenAI invites researchers to use their models, in fact giving at least 100,000 researchers free access[1], but there are also those that pay I've been thinking along exactly these lines... they very well could have a 21st century Mechanical Turk and its real superpower is getting people to "collaborate" asynchronously but it's just stealing their ideas and laundering them. I don't think it's purely that, of course... but "consult other clients' transcripts" would be an easy tool to write.
reasonableklout 22d ago
Can't both be true? 1. Systems that OpenAI is able to use (either public or private) are improving rapidly at open problems, even if they are still extraordinarily expensive 2. Researchers will inadvertently speed up the rate at which the AIs improve by feeding them valuable training data This is pretty much the definition of a data flywheel.
fwlr 22d ago
The relentless progress towards saturation of benchmarks is, I suspect, at least partly a similar story. Whatever holdout questions are used to evaluate GPT-x will be in the GPT-x conversation logs, and therefore in the training set for GPT-x+1.
- samuelknight 22d ago
  I can't think of better invention than one that can saturate a benchmark of every interesting problem.

Eddy_Viscosity2 23d ago

> they could be significantly piggybacking on human progress, This is AI in a nutshell, its a plagiarism machine. An abstraction layer between vast amounts of stolen human-generated data that filters out the liabilities and accountability for that original theft. Its an IP laundering system.

wiei 23d ago
That’s one perspective. I just view it as a thing that can brute force and produce outputs - that it has no way of ‘knowing’ - but doesn’t need to since it’s just running off of probability. No human can compete in that contest. But no llm can compete in the contest of ‘understanding’ and application in the real world - which is where 99% of the value is. I’m very pro AI long term btw but I’m not blinded.
- throwawayqqq11 22d ago
  Dont forget the holisitic validators/tools in the process. Probabilistics alone likely will not get you here. These rules are human made and without it, frontier models would not be able to compete, likely.
- foogazi 22d ago
  But it’s not brute force if it’s looking over everyone’s shoulder Brute force would have been solving Navier-Stokes in 88 hours after plagiarizing all known 20th century math When it needs to snoop live on what the actual mathematicians are working on that’s something else
  - wiei 22d ago
    No its happening whilst the human is working with it. The new inputs provided become part of the brute-force. This is what Scam Altman means by 'self-recursive'. Trust me I've seen it happen to myself. I no longer trust ChatGPT. I can see right through his act. Altman is one devious f8k.
- AnimalMuppet 22d ago
  AI needs humans to encode ideas in words. It needs those ideas to span the space of possibilities of, say, Navier Stokes. Then AI can be, as you say, a terrifyingly effective way to search that space. But when the building-block ideas are still being formed, I'm not sure that AI is good at forming them.
  - wiei 22d ago
    COrrect and this is how labour displacement happens. There are many actions being performed today that can be nicely packaged. Im already working on such a project.
robocat 22d ago
That's such an unquantifiable accusation. Plus it is an unfair standard since so many scientists in the past have been caught unethically using the work of others without attribution (and so many more have been accused). In history we also repeatedly see the phenomenon of multiple discovery or simultaneous invention. If that happens to AI because the topic is pregnant, would you call it "plagiarism" just to disparage AI? https://en.wikipedia.org/wiki/Multiple_discovery https://en.wikipedia.org/wiki/Multiple_discovery
- Eddy_Viscosity2 22d ago
  Your first example is the apt one here. In this case openAI was, allegedly, pilfering the work of the scientists into the AI. How is it an unfair standard. OpenAI stole the work of others to build the AI. That's not different than scientists stealing from other works as their own, or artists copying others work as their own, etc. It's all plagarism. I'm applying the same standard for everybody. As for multiple discovery, this is a thing, but I don't think the AI did a parallel discovery any more than Ray Kroc made the parallel discovery of the MacDonald brother's speedee service system.

wiei 23d ago

I’d argue the invitation of researchers was incredibly strategic. Sam Altman knows what he’s doing. He will happily screw these folks to one-up his competition.

JeremyNT 22d ago

> I've been wondering whether AI really is improving rapidly at open problems or we're being fooled. I think your suspicions are warranted and your explanation seems plausible. If better training data is the reason here, it would still be a case of the models doing something that is in and of itself super useful! The models really can take that data and distill it into solutions for similar problems faster than humans can. This is great! But there's so much vested interest in the AI companies to be opaque about all this, to hype up their models and avoid giving credit to people whose data made everything possible, that they would never tell us this fact if it were true. I feel like so much of the AI hype cycle is like this. The models develop extremely useful capabilities, but it's hard to understand what they really are through the hype. The lies and obfuscation by their owners who have vested interests in capturing the value they provide makes it impossible to take anything they say at face value.

YeGoblynQueenne 22d ago
>> If better training data is the reason here, it would still be a case of the models doing something that is in and of itself super useful! The models really can take that data and distill it into solutions for similar problems faster than humans can. This is great! It's perhaps great in the short term although it's not very clear who it's great for. I'm not sure mathematicians find it all so great, I mean. In the long term, if this contrives to destroy the tradition of human mathematics the whole endeavour is self-defeating. In time, there will be nobody left with the knowledge and skills to produce mathematics to train AI to do mathematics. And then we'll be left with no mathematics at all: we'll have no human mathematicians and no AI that can do mathematics, either.
- boothby 22d ago
  I was pretty depressed when I read about what happened with Navier Stokes this morning. The Clay Math prizes were a significant motivation through my math career, and I know a lot of computer scientists and physicists that feel similarly. I didn't think I was gonna resolve P vs NP or the BSD conjecture, but I did really research that felt like I was working towards something incredible. What is the younger generation left with? Hey kids betcha can't resolve the Collatz conjecture, our superintelligence can't either! Still pretty depressed about it, to be honest. Intellectualism is dead. We can return to happy agrarianism, I guess. At least the AI doesn't wanna eat my snap peas.

mikgp 22d ago

A mental model I was thinking about was - I remember when Travis Kalanick was talking about using the chatbot to discuss “vibe physics-ing” on the all-in podcast. And like - I think there’s a presumption you could make that AI models could overfit to asymptote towards just the capabilities and knowledge we currently have. And that would be amazing! And crazy useful. And there are probably a whole world of complex problems that remain unsolved because they’re adjacent to knowledge we have but they haven’t been invested in. But can a human reliably tell the difference between “can do 99.999% of the things we currently know how to do which includes a small subset of things we didn’t know we had the capacity to do” and “super intelligent math and science research pushing the frontier of what we know” A physicist that knows all the things we currently know in excruciating detail feels like it should be able to make the leap beyond the frontier. But since these are computer models it might just be that it can ride that line extraordinarily well while the line remains firm.

mannanj 22d ago

It tells me that AI companies are just another mechanism to extract and extort value from the masses for the rich. Just another rich man’s trick Perhaps the last one before they destroy that world and try to hide away as people forget and history is rewritten again. I don’t think they’ll succeed this time.

dgellow 22d ago
AI providers are pretty much the end boss of rent seeking, that’s for sure

glitchc 22d ago

The pudding is in the proof. The field is mathematics, the proof can be rigorously verified. If there is a flaw, OpenAI is out to lunch. If the proof is valid, OpenAI has produced something new.

amelius 22d ago
Did you read what they said? The question is now if OAI produced something new or just stole the researchers' good ideas.
- jsLavaGoat 22d ago
  Name one discovery ever that didn't depend on someone else's work.
  - amelius 22d ago
    Most discoveries did not happen by someone looking in someone else's notebooks without them knowing.
- glitchc 22d ago
  You seem to be unfamiliar about how research works. It's common to make an incremental advancement while citing prior work. The vast majority of papers out there fall into this bucket. Did the AI make incremental progress? Yes. Did it cite prior art? After some nudging, yes. It seems to me the academics are upset that AI scooped them. But scooping is a time-honored tradition between researchers. First to print and all that. In a nutshell, they are upset that they lost out on a publication. I will also point out for those unaware that any mathematics that is produced is automatically part of the public domain and can be used freely in derivative works. It is not a protected intellectual class like other works of art.
  - fg137 22d ago
    > But scooping is a time-honored tradition between researchers. Provided that it's properly accredited. And definitely not for others' unpublished work -- that's despised upon if not an academic integrity issue. People even point out that you should add a reference to certain papers during the peer review process.

bwfan123 22d ago

there are also attempts to crowdsource human research directions - like the caltech mathathon challenge : https://mathathonchallenge.com https://mathathonchallenge.com these would help models on the same problems at the expense of the researchers. basically, math researchers are the reverse centaurs but they dont realize it.

GPerson 22d ago
There is a very active open letter of over 1000 signatures from mathematicians in protest of this event. This event is targeting undergraduates. It previously suggested that math researchers already have no place in mathematics, and presents a limited and heavily distorted view of what mathematics research is.
- andrepd 22d ago
  I'm an AI skeptic, but I don't see how this squares with what the organisers of the event actually say. "It previously suggested that math researchers already have no place in mathematics"? I don't see this.
  - GPerson 22d ago
    The website previously said, “What is the role of a mathematician when AI can solve conjectures faster?” but they have removed it, possibly as a result of the letter since it happened after.
  - GPerson 22d ago
    Also I want to mention that the letter is not about AI skepticism, in any direct way at least.

agumonkey 22d ago

Seems easy to picture high stakes startup cutting corners to justify their fame.

YeGoblynQueenne 22d ago

>> Internal OpenAI models are reportedly solving open problems at a surprisingly fast rate[2] Maybe I'm failing to read that graph properly but the y axis says "pass rate" and it only goes up to 0.5. That would mean every single problem is at most half-solved. I don't know what that means though. What is "0.5 pass rate" in the context of "open math problems" (as in the graph title)?

red75prime 22d ago
I guess it's a fraction of problems on which a model produces a LEAN proof or a counterexample.
- YeGoblynQueenne 22d ago
  Wouldn't they just list the number of problems solved then?
  - dekhn 22d ago
    rates beat counts almost always.
    - YeGoblynQueenne 21d ago
      You're assuming too much from a single graph with vaguely named axes. I'll wait until there's more concrete information.

boothby 22d ago

> - OpenAI invites researchers to use their models, in fact giving at least 100,000 researchers free access[1], but there are also those that pay I've been thinking along exactly these lines... they very well could have a 21st century Mechanical Turk and its real superpower is getting people to "collaborate" asynchronously but it's just stealing their ideas and laundering them. I don't think it's purely that, of course... but "consult other clients' transcripts" would be an easy tool to write.

reasonableklout 22d ago

Can't both be true? 1. Systems that OpenAI is able to use (either public or private) are improving rapidly at open problems, even if they are still extraordinarily expensive 2. Researchers will inadvertently speed up the rate at which the AIs improve by feeding them valuable training data This is pretty much the definition of a data flywheel.

fwlr 22d ago

The relentless progress towards saturation of benchmarks is, I suspect, at least partly a similar story. Whatever holdout questions are used to evaluate GPT-x will be in the GPT-x conversation logs, and therefore in the training set for GPT-x+1.

samuelknight 22d ago
I can't think of better invention than one that can saturate a benchmark of every interesting problem.