5 ms·

> AGI isn't going to happen within the next 30 years so this is moot. The actual researchers have said so many times. It's only the business people and laypeopl

> AGI isn't going to happen within the next 30 years so this is moot. The actual researchers have said so many times. It's only the business people and laypeople whooping about AGI always being imminent. The statements of what "actual researchers" are you relying upon for your "next 30 years" estimate? How do you reconcile them with the sub-10- or even sub-5-years timelines of other AI researchers, like Daniel Kokotajlo[1] or Andrej Karpathy[2]? For that matter, what about polls of AI researchers, which usually obtain a median much shorter than 30 years [3]? [1] https://x.com/DKokotajlo/status/1991564542103662729 https://x.com/DKokotajlo/status/1991564542103662729 [2] https://x.com/karpathy/status/1980669343479509025 https://x.com/karpathy/status/1980669343479509025 [3] https://80000hours.org/2025/03/when-do-experts-expect-agi-to-arrive/ https://80000hours.org/2025/03/when-do-experts-expect-agi-to...

MadxX79 7mo ago
I'm guessing they have a lot of shares in the AI companies they work(ed) for, and they would like to pump their value so they can buy an even nicer carribean island than they can already afford?
- dwohnitmok 7mo ago
  Kokotajlo gave up all his shares in OpenAI as part of his refusal to sign a nondisparagement agreement with OpenAI.
- stratos123 7mo ago
  Kokotajlo in particular is notable for being the guy who quit OpenAI in 2024 in protest of their policy of requiring researchers to abide by a non-disparagement agreement to retain their equity. In the end OpenAI caved and changed their policy, but if he was lying all along to inflate the value of his shares, it would have been quite a 4d chess move of him to gamble the shares themselves on doing so.
  - MadxX79 7mo ago
    Isn't it just that he left way before gpt-5, then? At that point a sufficiently naive person could have believed that scaling was going to lead to AGI, but that sort of optimism died after he was already an outsider.
    - dwohnitmok 7mo ago
      Kokotajlo still believes we get AGI in the next few years. These are his most updated numbers at the moment: https://www.aifuturesmodel.com/ https://www.aifuturesmodel.com/
      - MadxX79 7mo ago
        I love the total lack of humility on that site. "What if the METR study turns out not to capture anything relevant? We just add a constant gap to be conservative!". But I guess these guys aren't really scientist, so it's probably a lot to ask that they relate critically to what they are doing and be honest about the limitations of their methods. What if it turns out that the more you scale the more your LLM resembles a lobotomized human. It looks like it goes really well in the beginning, but you are just never going to get to Einstein. How does that affect everything? What if it turned out that those AI companies were maybe having a whole bunch of humans solving the problems that are currently just below the 50% reliability threshold they set, and do fine tuning with those solutions. That will make their models perform better on the benchmark, but it's just training for the test... will the constant gap be a good approximation then?
  - dwohnitmok 7mo ago
    Not quite. Kokotajlo quit because he didn't think OpenAI would be good stewards of AGI (non-disparagement wasn't in the picture yet). As part of his exit OpenAI asked him to sign a non-disparagement as a condition of keeping his equity. He refused and gave up his equity. To the best of my knowledge he lost that equity permanently and no longer has any stake in OpenAI (even if this episode later led to an outcry against OpenAI causing them to remove the non-disparagement agreement from future exits).
Spivak 7mo ago
See this is a fun game because when you're fishing for a breakthrough you can predict tomorrow or 100 years. Nobody, even experts have any idea until it happens and they're holding it in their hands. To have any kind of accurate prediction you would have to have already observed other civilizations discover AGI to say how close the environment is to even be capable of making the leap. We could be missing something huge, we could need multiple seemingly unrelated breakthroughs to get there. We're for sure closer, but we could still be miles away, GPTs might even barking up the wrong tree. Why this discussion is already annoying and poised to get so much worse is because now hundred billion dollar companies have a direct financial incentive to say they did it so I expect the definition will get softened to near meaninglessness so some marketing department can slap AGI on their thing.
linkregister 7mo ago
I think you are overindexing on the integer value given in the parent post, rather than seeing the essence that LLMs in their current form only excel on tasks they have been specifically trained for. Karpathy himself has publicly stated that AGI itself is only possible with a new paradigm (that his group is working toward). He claims RHLF and attention models are near the end of their logarithmic curve. The concept of the "self-training AI" is likely impossible without a new kind of model. We will likely see some classes of human skills completely taken over by LLMs this decade: call centers (already capable in 2026), SWE (the next couple years). Bear in mind the frontier labs have spend many billions on exhaustive training on every aspect of these domains. They are focusing training on the highest value occupations, but the long tail is huge. It will be interesting to see if this investment will be obviated by a "real AGI" capable of learning without going through the capital-intensive training steps of current models.
- stratos123 7mo ago
  Personally I'm not even sold on the current paradigm being too limited to produce AGI - there are still several OOMs worth of compute increase available, plus the algorithmic improvements have overall been accumulating faster than predicted. But even assuming that a major breakthrough is required, it seems ludicrous to me to go from that to a timeline of a decade or more. This isn't like fusion power research, where you spend 10 years building a new installation only to find new problems. Software development is inherently faster, and AI research in particular has been moving extremely quickly in the past. (GPT-3 is only 6 years old.) I don't think a wall in AI progress, if one comes at all, will last more than a few years.
  - paulryanrogers 7mo ago
    > there are still several OOMs worth of compute increase available Where and how? Aren't we reaching the physical limits of making transistors smaller?
  - deleted 7mo ago
    [deleted]
  - 7mo ago

MadxX79 7mo ago

I'm guessing they have a lot of shares in the AI companies they work(ed) for, and they would like to pump their value so they can buy an even nicer carribean island than they can already afford?

dwohnitmok 7mo ago
Kokotajlo gave up all his shares in OpenAI as part of his refusal to sign a nondisparagement agreement with OpenAI.
stratos123 7mo ago
Kokotajlo in particular is notable for being the guy who quit OpenAI in 2024 in protest of their policy of requiring researchers to abide by a non-disparagement agreement to retain their equity. In the end OpenAI caved and changed their policy, but if he was lying all along to inflate the value of his shares, it would have been quite a 4d chess move of him to gamble the shares themselves on doing so.
- MadxX79 7mo ago
  Isn't it just that he left way before gpt-5, then? At that point a sufficiently naive person could have believed that scaling was going to lead to AGI, but that sort of optimism died after he was already an outsider.
  - dwohnitmok 7mo ago
    Kokotajlo still believes we get AGI in the next few years. These are his most updated numbers at the moment: https://www.aifuturesmodel.com/ https://www.aifuturesmodel.com/
    - MadxX79 7mo ago
      I love the total lack of humility on that site. "What if the METR study turns out not to capture anything relevant? We just add a constant gap to be conservative!". But I guess these guys aren't really scientist, so it's probably a lot to ask that they relate critically to what they are doing and be honest about the limitations of their methods. What if it turns out that the more you scale the more your LLM resembles a lobotomized human. It looks like it goes really well in the beginning, but you are just never going to get to Einstein. How does that affect everything? What if it turned out that those AI companies were maybe having a whole bunch of humans solving the problems that are currently just below the 50% reliability threshold they set, and do fine tuning with those solutions. That will make their models perform better on the benchmark, but it's just training for the test... will the constant gap be a good approximation then?
- dwohnitmok 7mo ago
  Not quite. Kokotajlo quit because he didn't think OpenAI would be good stewards of AGI (non-disparagement wasn't in the picture yet). As part of his exit OpenAI asked him to sign a non-disparagement as a condition of keeping his equity. He refused and gave up his equity. To the best of my knowledge he lost that equity permanently and no longer has any stake in OpenAI (even if this episode later led to an outcry against OpenAI causing them to remove the non-disparagement agreement from future exits).

Spivak 7mo ago

See this is a fun game because when you're fishing for a breakthrough you can predict tomorrow or 100 years. Nobody, even experts have any idea until it happens and they're holding it in their hands. To have any kind of accurate prediction you would have to have already observed other civilizations discover AGI to say how close the environment is to even be capable of making the leap. We could be missing something huge, we could need multiple seemingly unrelated breakthroughs to get there. We're for sure closer, but we could still be miles away, GPTs might even barking up the wrong tree. Why this discussion is already annoying and poised to get so much worse is because now hundred billion dollar companies have a direct financial incentive to say they did it so I expect the definition will get softened to near meaninglessness so some marketing department can slap AGI on their thing.

linkregister 7mo ago

I think you are overindexing on the integer value given in the parent post, rather than seeing the essence that LLMs in their current form only excel on tasks they have been specifically trained for. Karpathy himself has publicly stated that AGI itself is only possible with a new paradigm (that his group is working toward). He claims RHLF and attention models are near the end of their logarithmic curve. The concept of the "self-training AI" is likely impossible without a new kind of model. We will likely see some classes of human skills completely taken over by LLMs this decade: call centers (already capable in 2026), SWE (the next couple years). Bear in mind the frontier labs have spend many billions on exhaustive training on every aspect of these domains. They are focusing training on the highest value occupations, but the long tail is huge. It will be interesting to see if this investment will be obviated by a "real AGI" capable of learning without going through the capital-intensive training steps of current models.

stratos123 7mo ago
Personally I'm not even sold on the current paradigm being too limited to produce AGI - there are still several OOMs worth of compute increase available, plus the algorithmic improvements have overall been accumulating faster than predicted. But even assuming that a major breakthrough is required, it seems ludicrous to me to go from that to a timeline of a decade or more. This isn't like fusion power research, where you spend 10 years building a new installation only to find new problems. Software development is inherently faster, and AI research in particular has been moving extremely quickly in the past. (GPT-3 is only 6 years old.) I don't think a wall in AI progress, if one comes at all, will last more than a few years.
- paulryanrogers 7mo ago
  > there are still several OOMs worth of compute increase available Where and how? Aren't we reaching the physical limits of making transistors smaller?
- deleted 7mo ago
  [deleted]
- 7mo ago