6 ms·

I don't think you want to bring cost into this argument. Even if the cost was $1 mil for these 10 problems, that's maybe 10-20 math researchers for a year. Do

I don't think you want to bring cost into this argument. Even if the cost was $1 mil for these 10 problems, that's maybe 10-20 math researchers for a year. Do you really think that if you paid that to humans, they will deliver the same results?

mungaihaha 3mo ago
Grad students on zero pay solve problems like this everyday. What exactly is your point here?
- mirzap 3mo ago
  Even if they can solve problems like this every day, you still have a very limited number of grad students who can solve them. With model capabilities like this, you can have the equivalent of millions of grad students who can solve problems like this.
- whattheheckheck 3mo ago
  Give the grad students these resources and they can do even more!!!
- gbnwl 3mo ago
  Everyday? Which 10 problems were solved by mathematics grad students in the past 10 days? OK I’ll grant that it’s not your obligation to be my search function (despite you making the wild assertion in the first place), so instead can you just point us to the latest grad student solved problem of this level that you know of?
  - gowld 3mo ago
    [dead]
- r0uv3n 3mo ago
  Grad students do not solve problems such as the existence of non-sofic groups every day.
  - mungaihaha 3mo ago
    Plenty of 'advances in mathematics' done pre-llm, no?
- maleldil 3mo ago
  Zero pay? These would be PhD candidates; surely they have a stipend?
  - Readerium 3mo ago
    Nopes, often times especially in math they get paid due to teaching duties (at least in the US). So technically for the math research part they are not getting any stipend.
uh_uh 3mo ago
It is comical at this point. Some people just can not stand the thought of AI actually delivering and are trying to find whatever ways to discredit it.
- vector_spaces 3mo ago
  Sure, but I don't really understand what the argument is to _not_ be transparent about methodology, since if the models are so powerful, then doing so would easily support the claims and put these concerns to rest. People are right to be skeptical given what is being implied and the orientation of the narrative I know it's more exciting to say "AI disproved a longstanding conjecture" vs to say "it did so AND it took several PhD specialists in the field this many attempts to even produce a prompt that got the model spitting out something useful under some configurations, and many iterations to optimize the configurations, and the prompt itself, and many trials with that configuration to solve the problem. All told we spent more than a typical math academic can hope make in their career." By not being transparent, they invite skepticism and cynical takes, like maybe it's just that tempered and qualified claims are an existential threat to companies that are fully subsidized by the hype train? I don't know. Either way, it seems like it would be easy to address these, so why should they not do it? To be clear, even if that tempered version is close to reality, it doesn't make the models not useful! It just forces a certain calibration of expectations I say this btw as someone who uses these things extensively, including to disprove an old conjecture my advisor and I were stuck on recently. I know they are powerful and that everything is different now because of them. Let's be sober when discussing them though
- dgacmu 3mo ago
  This isn't really about delivering - it's more about helping to understand the shape of problems that AI can solve right now. If they took 1000 problems and threw the model at it and it solved these ten, is there something we learn about these ten problems and the kinds of things that current AI is good at? That's very different from picking ten problems _at random_ and solving all of them successfully, which would suggest a much less bumpy capability surface. It's interesting and it would be good science to release it.
kevinwang 3mo ago
It would still provide better context to see the numbers that the parent proposes, though.
robotpepi 3mo ago
it's still important. not everyone has access to 1 million USD. saying it "only" coat 2000 USD is highly misleading for the discussion and future. the concentration of power is a huge problem with AI.
tchalla 3mo ago
Mentioning cost is fine, comparing may not be.
wbl 3mo ago
If you told them this was the problem and they would still have a job if they failed probably. The reasons people don't go head on these problems is career incentives and psychology.
fasterik 3mo ago
You need to bring both cost and benefit into the argument, and it's not necessarily an obvious win for either side. There are a few complicating factors here. The cost of running a model is not only $/token, but the salaries of the people managing/orchestrating the models, deciding what theorems to try, etc. Once we factor that in, how much are we really paying per theorem? The other factor is the subjective component of the value of a theorem. Not all theorems are created equal, and the only way to really measure the value is to ask professional mathematicians for their opinion, or publish the results and look at citations over months/years. Once we have both of these nailed down, then we can start to do the cost/benefit analysis. To be fair, we should actually compare three groups: human experts, hybrid agent/human expert teams, and fully autonomous agents.

mungaihaha 3mo ago

Grad students on zero pay solve problems like this everyday. What exactly is your point here?

mirzap 3mo ago
Even if they can solve problems like this every day, you still have a very limited number of grad students who can solve them. With model capabilities like this, you can have the equivalent of millions of grad students who can solve problems like this.
whattheheckheck 3mo ago
Give the grad students these resources and they can do even more!!!
gbnwl 3mo ago
Everyday? Which 10 problems were solved by mathematics grad students in the past 10 days? OK I’ll grant that it’s not your obligation to be my search function (despite you making the wild assertion in the first place), so instead can you just point us to the latest grad student solved problem of this level that you know of?
- gowld 3mo ago
  [dead]
r0uv3n 3mo ago
Grad students do not solve problems such as the existence of non-sofic groups every day.
- mungaihaha 3mo ago
  Plenty of 'advances in mathematics' done pre-llm, no?
maleldil 3mo ago
Zero pay? These would be PhD candidates; surely they have a stipend?
- Readerium 3mo ago
  Nopes, often times especially in math they get paid due to teaching duties (at least in the US). So technically for the math research part they are not getting any stipend.

uh_uh 3mo ago

It is comical at this point. Some people just can not stand the thought of AI actually delivering and are trying to find whatever ways to discredit it.

vector_spaces 3mo ago
Sure, but I don't really understand what the argument is to _not_ be transparent about methodology, since if the models are so powerful, then doing so would easily support the claims and put these concerns to rest. People are right to be skeptical given what is being implied and the orientation of the narrative I know it's more exciting to say "AI disproved a longstanding conjecture" vs to say "it did so AND it took several PhD specialists in the field this many attempts to even produce a prompt that got the model spitting out something useful under some configurations, and many iterations to optimize the configurations, and the prompt itself, and many trials with that configuration to solve the problem. All told we spent more than a typical math academic can hope make in their career." By not being transparent, they invite skepticism and cynical takes, like maybe it's just that tempered and qualified claims are an existential threat to companies that are fully subsidized by the hype train? I don't know. Either way, it seems like it would be easy to address these, so why should they not do it? To be clear, even if that tempered version is close to reality, it doesn't make the models not useful! It just forces a certain calibration of expectations I say this btw as someone who uses these things extensively, including to disprove an old conjecture my advisor and I were stuck on recently. I know they are powerful and that everything is different now because of them. Let's be sober when discussing them though
dgacmu 3mo ago
This isn't really about delivering - it's more about helping to understand the shape of problems that AI can solve right now. If they took 1000 problems and threw the model at it and it solved these ten, is there something we learn about these ten problems and the kinds of things that current AI is good at? That's very different from picking ten problems _at random_ and solving all of them successfully, which would suggest a much less bumpy capability surface. It's interesting and it would be good science to release it.

kevinwang 3mo ago

It would still provide better context to see the numbers that the parent proposes, though.

robotpepi 3mo ago

it's still important. not everyone has access to 1 million USD. saying it "only" coat 2000 USD is highly misleading for the discussion and future. the concentration of power is a huge problem with AI.

tchalla 3mo ago

Mentioning cost is fine, comparing may not be.

wbl 3mo ago

If you told them this was the problem and they would still have a job if they failed probably. The reasons people don't go head on these problems is career incentives and psychology.

fasterik 3mo ago

You need to bring both cost and benefit into the argument, and it's not necessarily an obvious win for either side. There are a few complicating factors here. The cost of running a model is not only $/token, but the salaries of the people managing/orchestrating the models, deciding what theorems to try, etc. Once we factor that in, how much are we really paying per theorem? The other factor is the subjective component of the value of a theorem. Not all theorems are created equal, and the only way to really measure the value is to ask professional mathematicians for their opinion, or publish the results and look at citations over months/years. Once we have both of these nailed down, then we can start to do the cost/benefit analysis. To be fair, we should actually compare three groups: human experts, hybrid agent/human expert teams, and fully autonomous agents.