4 ms·

Not being an expert in any of the fields OpenAI has "advanced" I don't want to prematurely downplay the significance of this contribution. However, I am worried

by kcexn 2mo ago

Not being an expert in any of the fields OpenAI has "advanced" I don't want to prematurely downplay the significance of this contribution. However, I am worried that the language they are using in this blog post is exaggerating for the sake of marketing. It is true there hasn't been a reliable computational approach to solving these problems before. But do these proofs contribute new ideas to the mathematical corpus, or are they simply an effective method to exhaustively search the literature for the right combination of existing tools to apply to the problem? Essentially, did these problems seem like they had an intuitive answer and were feasible to prove before, just not high enough value targets for an expert to invest time into? Or were they fundamentally difficult prior to this point and it appears that AI has done something more than just throw the problem into a big solver.

simianwords 2mo ago
> However, I am worried that the language they are using in this blog post is exaggerating for the sake of marketing Your worry.... is because they used the word advanced? For marketing? The word is used very appropriately here. There were PhD's who spent a big part of their career tackling these problems.
- kcexn 2mo ago
  I have no idea how many PhD's have spent how much time of their careers tackling these very specific problems, and I doubt you do either. I'm trying to understand if these specific problems were the kinds of problems that would have justified an expert investing weeks or months to solve. Or if they were the kinds of problems that would normally have been given to students to investigate.
  - hollowcelery 2mo ago
    They are significant problems which experts have spent months or years studying. I heard a mathematician say that resolving non-sofic groups and Connes's rigidity would be career-defining for a mathematician.
Ar-Curunir 2mo ago
The problems from CS (CVP and circuit complexity) are very important problems that have been worked on by top researchers for 30-40 years. Some of these researchers include Turing Award winners. A solution to them would be a best-paper award at many top CS conferences.
- kcexn 2mo ago
  I assume you're talking about No. 5, the arithmetic circuit complexity bound? The existence of a lower bound than state-of-the-art is certainly a significant result and worth publishing. But the wording of the result makes it sound like we don't know what the lowest possible complexity bound might be. So, prior to this result did we think there couldn't be a lower possible bound? Or did the arithmetic circuit community think there were lower possible bounds but didn't see it as a high value target for experts to tackle (maybe a problem that was instead regularly given to students to study).
  - Ar-Curunir 2mo ago
    Circuit complexity lower bounds (and lower bounds in general) are notoriously difficult to come across. For example, despite our best efforts, the state of the art lower bounds on time complexity of algorithms for solving 3SAT is O(n). In contrast, our best algorithms for the task run in time roughly O(2^n). That’s an exponential gap. This is despite decades of trying to find lower bounds.
    - ninkendo 2mo ago
      > the state of the art lower bounds on time complexity of algorithms for solving 3SAT is O(n) Wow, that’s pretty stark. “What’s the minimum time it would take to solve this problem?” “Well, at the very least you’d have to read the input the whole way through”
patcon 2mo ago
"Breakthrough research" can be defined (in the citation record) as research that both (1) becomes highly cited, and (2) brings together citation chains that were previously not showing up together. Mundane incremental research is cobbled from existing citations that already appear nearby in the record. Basically, innovative research is a measure of bridging thought and domains that were previously not bridged. It's quite concrete as a measure in the citation record. So we can know pretty conclusively. Puja Ohlhaver gave a talk on this[1], and ran some experiments (that I had the pleasure to support on) [1]: https://www.youtube.com/watch?v=guLDNMAOn24 https://www.youtube.com/watch?v=guLDNMAOn24
- casey2 2mo ago
  Breakthrough math research is very rarely highly cited. Maybe some combination of pretraining scale, inference speed and orchestration will help, but it's telling that OpenAI is solving random math research problems rather than bedrock algorithms and their implementation. Even as cool as the tech is, there still is very much a clock that they have to outrace before they collapse.
- kcexn 2mo ago
  I'm not arguing that this isn't innovative or worthy of publication. Basically any result that moves the needle meets those criteria. I'm interested in how the results that OpenAI has published here differs from finding optimality solutions for incredibly niche optimization problems by throwing the problem in an enormous solver.
  - sally_glance 2mo ago
    They differ in that there hasn't been a solver you could have thrown them at. I guess you could argue their harness + LLM setup is a "solver", but the approach is so different from what we used that word for in the past that I don't think it would be appropriate.
QuesnayJr 2mo ago
The ones I'm familiar with are big breakthroughs, but they are both counterexamples. Examples have an advantage in that once you have the example in hand and a sketch of the proof (which they have provided), then an expert can probably work out the details themselves. The sofic groups question was the outstanding question about sofic groups. Almost everyone thought that non-sofic groups existed, and there were plausible candidates, but proving a group was non-sofic was out of reach. Now that we know how to do it once, we can probably do it a lot more. The Connes rigidity conjecture I think people thought was false, but it was a provocative claim to make. The significance of conjectures is frequently not that the answer to the question is "yes", but that we don't know how to answer the question. And now, apparently, we do.
- kcexn 2mo ago
  Interesting. Do you have any more specific insights into where you feel AI was a big value-add to these problems? I don't want to be overly dismissive of AI, but I also feel that the AI hype engine frequently positions claims as being 'ground-breaking' when they are really just interesting incremental results. The general consensus of developers is that AI can only do the work of a strong 'junior'. Yet as soon as we are presented with pure mathematical results, people seem incredibly ready to accept that AI can do more than what a strong student could achieve.
  - QuesnayJr 2mo ago
    They are more than a strong student could achieve. I'm not equally familiar with the problems, but the ones I'm familiar with, if a student solved them people would be thinking "that's someone on track to win the Fields Medal one day". If it works better here than for programming, then I would guess it's because you can give it a very precise prompt, so you either solve the problem or you don't. If you read the prompts people have shared for problems like this, then the instructions are basically "Solve this problem. Don't give up early. Don't solve a similar problem."
- robotpepi 2mo ago
  > but proving a group was non-sofic was out of reach a colleague was telling me that the base idea for proving that something is not sofic already appeared in the literature around 2019 or so (this is the "expanders graphs" that are mentioned in OpenAI s paper. no one had managed to find a concrete example though. this doesn't make the result less impressive in any case.
x0mej 2mo ago
You’ve received the expert answer several times. You just don’t seem to like the answer.
- QwenGlazer9000 2mo ago
  Forgive me for taking everything salesmen say with a grain of salt.
  - deleted 2mo ago
    [deleted]
bamboozled 2mo ago
It’s a marketing. They are a sham company. If this article was by Scientific American or something it would be worth a lot more. They are literally trying to keep the hype train on track. Also on HN front page today: AI's debt binge can't last, hidden borrowing reaches $1.65T (fortune.com) https://news.ycombinator.com/item?id=49160699 https://news.ycombinator.com/item?id=49160699
- qbit42 2mo ago
  It is marketing, they are a shady company, and yet, if someone had access to these results before today's modern AI tools, they could get tenure at any university in the world.
  - bamboozled 2mo ago
    As others have said, it's hard to know how significant these results are without more transparency around the methods used to obtain them.
    - azan_ 2mo ago
      Not really. Why would that be the case?
      - camdenreslink 2mo ago
        If the prompt was crafted by an expert mathematician with pages of context and insights provided to the LLM to put it on the right path, that is a lot less impressive than a prompt that just says "find a counterexample to this conjecture". Similarly, if they are spending millions of dollars in inference just churning on thousands of problems and these are the 10 solutions they came up with, that would be less impressive than if they chose these 10 problems specifically and were able to come up with solutions.

by kcexn 2mo ago

simianwords 2mo ago

> However, I am worried that the language they are using in this blog post is exaggerating for the sake of marketing Your worry.... is because they used the word advanced? For marketing? The word is used very appropriately here. There were PhD's who spent a big part of their career tackling these problems.

kcexn 2mo ago
I have no idea how many PhD's have spent how much time of their careers tackling these very specific problems, and I doubt you do either. I'm trying to understand if these specific problems were the kinds of problems that would have justified an expert investing weeks or months to solve. Or if they were the kinds of problems that would normally have been given to students to investigate.
- hollowcelery 2mo ago
  They are significant problems which experts have spent months or years studying. I heard a mathematician say that resolving non-sofic groups and Connes's rigidity would be career-defining for a mathematician.

Ar-Curunir 2mo ago

The problems from CS (CVP and circuit complexity) are very important problems that have been worked on by top researchers for 30-40 years. Some of these researchers include Turing Award winners. A solution to them would be a best-paper award at many top CS conferences.

kcexn 2mo ago
I assume you're talking about No. 5, the arithmetic circuit complexity bound? The existence of a lower bound than state-of-the-art is certainly a significant result and worth publishing. But the wording of the result makes it sound like we don't know what the lowest possible complexity bound might be. So, prior to this result did we think there couldn't be a lower possible bound? Or did the arithmetic circuit community think there were lower possible bounds but didn't see it as a high value target for experts to tackle (maybe a problem that was instead regularly given to students to study).
- Ar-Curunir 2mo ago
  Circuit complexity lower bounds (and lower bounds in general) are notoriously difficult to come across. For example, despite our best efforts, the state of the art lower bounds on time complexity of algorithms for solving 3SAT is O(n). In contrast, our best algorithms for the task run in time roughly O(2^n). That’s an exponential gap. This is despite decades of trying to find lower bounds.
  - ninkendo 2mo ago
    > the state of the art lower bounds on time complexity of algorithms for solving 3SAT is O(n) Wow, that’s pretty stark. “What’s the minimum time it would take to solve this problem?” “Well, at the very least you’d have to read the input the whole way through”

patcon 2mo ago

"Breakthrough research" can be defined (in the citation record) as research that both (1) becomes highly cited, and (2) brings together citation chains that were previously not showing up together. Mundane incremental research is cobbled from existing citations that already appear nearby in the record. Basically, innovative research is a measure of bridging thought and domains that were previously not bridged. It's quite concrete as a measure in the citation record. So we can know pretty conclusively. Puja Ohlhaver gave a talk on this[1], and ran some experiments (that I had the pleasure to support on) [1]: https://www.youtube.com/watch?v=guLDNMAOn24 https://www.youtube.com/watch?v=guLDNMAOn24

casey2 2mo ago
Breakthrough math research is very rarely highly cited. Maybe some combination of pretraining scale, inference speed and orchestration will help, but it's telling that OpenAI is solving random math research problems rather than bedrock algorithms and their implementation. Even as cool as the tech is, there still is very much a clock that they have to outrace before they collapse.
kcexn 2mo ago
I'm not arguing that this isn't innovative or worthy of publication. Basically any result that moves the needle meets those criteria. I'm interested in how the results that OpenAI has published here differs from finding optimality solutions for incredibly niche optimization problems by throwing the problem in an enormous solver.
- sally_glance 2mo ago
  They differ in that there hasn't been a solver you could have thrown them at. I guess you could argue their harness + LLM setup is a "solver", but the approach is so different from what we used that word for in the past that I don't think it would be appropriate.

QuesnayJr 2mo ago

The ones I'm familiar with are big breakthroughs, but they are both counterexamples. Examples have an advantage in that once you have the example in hand and a sketch of the proof (which they have provided), then an expert can probably work out the details themselves. The sofic groups question was the outstanding question about sofic groups. Almost everyone thought that non-sofic groups existed, and there were plausible candidates, but proving a group was non-sofic was out of reach. Now that we know how to do it once, we can probably do it a lot more. The Connes rigidity conjecture I think people thought was false, but it was a provocative claim to make. The significance of conjectures is frequently not that the answer to the question is "yes", but that we don't know how to answer the question. And now, apparently, we do.

kcexn 2mo ago
Interesting. Do you have any more specific insights into where you feel AI was a big value-add to these problems? I don't want to be overly dismissive of AI, but I also feel that the AI hype engine frequently positions claims as being 'ground-breaking' when they are really just interesting incremental results. The general consensus of developers is that AI can only do the work of a strong 'junior'. Yet as soon as we are presented with pure mathematical results, people seem incredibly ready to accept that AI can do more than what a strong student could achieve.
- QuesnayJr 2mo ago
  They are more than a strong student could achieve. I'm not equally familiar with the problems, but the ones I'm familiar with, if a student solved them people would be thinking "that's someone on track to win the Fields Medal one day". If it works better here than for programming, then I would guess it's because you can give it a very precise prompt, so you either solve the problem or you don't. If you read the prompts people have shared for problems like this, then the instructions are basically "Solve this problem. Don't give up early. Don't solve a similar problem."
robotpepi 2mo ago
> but proving a group was non-sofic was out of reach a colleague was telling me that the base idea for proving that something is not sofic already appeared in the literature around 2019 or so (this is the "expanders graphs" that are mentioned in OpenAI s paper. no one had managed to find a concrete example though. this doesn't make the result less impressive in any case.

x0mej 2mo ago

You’ve received the expert answer several times. You just don’t seem to like the answer.

QwenGlazer9000 2mo ago
Forgive me for taking everything salesmen say with a grain of salt.
- deleted 2mo ago
  [deleted]

bamboozled 2mo ago

It’s a marketing. They are a sham company. If this article was by Scientific American or something it would be worth a lot more. They are literally trying to keep the hype train on track. Also on HN front page today: AI's debt binge can't last, hidden borrowing reaches $1.65T (fortune.com) https://news.ycombinator.com/item?id=49160699 https://news.ycombinator.com/item?id=49160699

qbit42 2mo ago
It is marketing, they are a shady company, and yet, if someone had access to these results before today's modern AI tools, they could get tenure at any university in the world.
- bamboozled 2mo ago
  As others have said, it's hard to know how significant these results are without more transparency around the methods used to obtain them.
  - azan_ 2mo ago
    Not really. Why would that be the case?
    - camdenreslink 2mo ago
      If the prompt was crafted by an expert mathematician with pages of context and insights provided to the LLM to put it on the right path, that is a lot less impressive than a prompt that just says "find a counterexample to this conjecture". Similarly, if they are spending millions of dollars in inference just churning on thousands of problems and these are the 10 solutions they came up with, that would be less impressive than if they chose these 10 problems specifically and were able to come up with solutions.