7 ms·

This is comparable to another research/estimate with similar findings at https://ai-2027.com/ https://ai-2027.com/ I find the proposed timelines aggressive (~AG

by e1g 1y ago

This is comparable to another research/estimate with similar findings at https://ai-2027.com/ https://ai-2027.com/ I find the proposed timelines aggressive (~AGI in ~3 years), but the people behind this thinking are exceptionally thoughtful and well-versed in all related fields.

baq 1y ago
exponential curves tend to feel linear early and obviously non-linear in hindsight. add to that extreme dependence on starting conditions and you get a perfect mix of incompatibility what human psychology. this is why it's all so scary. almost nobody believes it'll happen until it's basically already happened or can't be stopped.
- XorNot 1y ago
  And sigmoidal curves feel exponential at the start, then linear. I see little evidence we're headed towards an exponential regime: the cost and resource usage versus capability hasn't been acting that way.
  - baq 1y ago
    OTOH we know there are multiple orders of magnitude of efficiency to gain: the brain does what it does at 20W. Forecasting where the inflection point is on the sigmoid is as hard as everything else.
    - XorNot 1y ago
      While true, I'd argue that's worse for current progress: the overall trend has been to throw mammoth amounts of electricity at the problem via compute. In so much as something like DeepSeek proves there's gains to be made, the current growth in resource usage would speak against declaring it proves improvement soon. Like plug that issue into known hardware limitations due to physical limits (i.e. we're already up against the wall re: feature sizes) and I'm even more skeptical. If we were looking at a buy down in training resources which was accelerating, then I'd be a lot more interested.
refulgentis 1y ago
Is the slatestarcodex guy "well-versed in all related fields"? Isn't he a psychologist? What would being well versed in all related fields even mean? Especially in the context of the output, a fictional overthetop geopolitics text that leaves the AI stuff at "at timestamp N+1, the model gets better" It's of the same stuff as fan fiction, layers of odd geopolitical stuff, no science fiction. Even at that it is internally incoherent quite regularly (the White House looks to jail the USA Champion AI Guy for some reason while they're in the midst of declaring it an existential war against China) Titillating, in that Important Things are happening. Sophomoric, in that the important things are off camera and an excuse to talk about something else. I say that as someone who believes people 20 years from now will say it happened somewhere between sonnets agentic awareness and o3s uncanny post human ability to turn a factual inquiry about the ending of a TV show into an incisive therapy session
- e1g 1y ago
  The prime mover behind this project is Daniel Kokotajlo, an ex-OpenAI researcher who documented his last predictions in 2021 [1], and much of that essay turned out to be nearly prophetic. Scott Alexander is a psychiatrist, but more relevant is that he dedicated the last decade to thinking and writing about societal forces, which is useful when forecasting AI. Other contributors are professional AI researchers and forecasters. [1] https://www.lesswrong.com/posts/6Xgy6CAf2jqHhynHL/what-2026-looks-like https://www.lesswrong.com/posts/6Xgy6CAf2jqHhynHL/what-2026-...
  - refulgentis 1y ago
    Oh my. I had no idea until now, that was exactly the same flavor and apparently, this is no coincidence. I'm not sure it was prophetic, it was a good survey of the field, but the claim was...a plot of grade schooler to PhD against year. I'm glad he got a paycheck from OpenAI at one point in time. I got one from Google in one point in time. Both of these projects are puffery, not scientific claims of anything, or claims of anything at all other than "at timestamp N+1, AI will be better than timestamp N, on an exponential curve" Utterly bog-standard boring claim going back to 2016 AFAIK. Not the product of considered expertise. Not prophetic.
ajb 1y ago
There's no indication that any of them are well versed in anything to do with the physical world (manufacturing, electronics, agriculture, etc) but they forecast that AI can replace the human physical economy in a few years, including manufacturing it's own chips.
- popcorncowboy 1y ago
  Because 1, 2, skip a few, 99, ASI. Check, and mate.
- lordofgibbons 1y ago
  > but they forecast that AI can replace the human physical economy in a few years I guess it depends how many years YOU mean. They're absolutely not claiming that there will be armies of robots making chips in 3 years. They're claiming there will be some semblance of AGI that will be capable of improving/speeding-up the AI development loop within 3 years.
  - ajb 1y ago
    The ai-2027 folks absolutely do claim that. Their scenario has "Humans realize that they are obsolete" in late 2029.
  - pjc50 1y ago
    > They're absolutely not claiming that there will be armies of robots making chips in 3 years. They're claiming there will be some semblance of AGI that will be capable of improving/speeding-up the AI development loop within 3 years Motte and bailey: huge claim in the headline, little tiny claim in the body.
amarcheschi 1y ago
That's more of a blog article than a research paper... Scott Alexander (one of the writers), yudkowsky, and the others (not the other authors, the other group of "thinkers" with similar ideas) are more or less Ai doomers with no actual background in machine learning/ai I don't think why we should listen to them. Especially when that blog page is formatted in a deceptive way to look like a research paper It's not science, it's science fiction
- ben_w 1y ago
  > are more or less Ai doomers with no actual background in machine learning/ai I don't think why we should listen to them. Weather vs. climate. The question they're asking isn't about machine learning specifically, it's about the risks of generic optimisers optimising a utility function, and the difficulty of specifying a utility function in a way that doesn't have unfortunate side effects. The examples they give also work with biology (genetics and the difference between what your genes "want" and what your brain "wants") and with governance (laws and loopholes, cobra effects, etc.). This is why a lot (I don't want to say "majority") of people who do have an actual background in machine learning and AI, pay attention to doomer arguments. Some of them* may be business leaders using the same language to BS their way into regulatory capture, but my experience of "real" AI researchers is they're mostly also "safety is important, Yudkowsky makes good points about XYZ" even if they would also say "my P(doom) is only 10%, not 95% like Yudkowsky". * I'm mainly thinking of Musk here, thanks to him saying "AI is summoning the demon" while also having an AI car company, funding OpenAI in the early years and now being in a legal spat with it that looks like it's "hostile takeover or interfere to the same end", funding another AI company, building humanoid robots and showing off ridiculous compute hardware, having brain implant chips, etc.
  - amarcheschi 1y ago
    >The question they're asking isn't about machine learning specifically, it's about the risks of generic optimisers optimising a utility function, and the difficulty of specifying a utility function in a way that doesn't have unfortunate side effects. The examples they give also work with biology (genetics and the difference between what your genes "want" and what your brain "wants") and with governance (laws and loopholes, cobra effects, etc.). But you do need some kind of base knowledge, if you want to talk about this. Otherwise you're saying "what if we create God". And last time I checked it wasn't possible. And what's with the existential risk obsession? That's like a bad retelling of the Pascal bet on the existence of God. I'm relieved that at least in italy I still have to find someone in Ai taking them into consideration for more than a few minutes during an ethics course (with students sneering at the ideas of bostrom possible futures), and again, it's held by a professor with no technical knowledge with whom i often disagree due to this

by e1g 1y ago

baq 1y ago

exponential curves tend to feel linear early and obviously non-linear in hindsight. add to that extreme dependence on starting conditions and you get a perfect mix of incompatibility what human psychology. this is why it's all so scary. almost nobody believes it'll happen until it's basically already happened or can't be stopped.

XorNot 1y ago
And sigmoidal curves feel exponential at the start, then linear. I see little evidence we're headed towards an exponential regime: the cost and resource usage versus capability hasn't been acting that way.
- baq 1y ago
  OTOH we know there are multiple orders of magnitude of efficiency to gain: the brain does what it does at 20W. Forecasting where the inflection point is on the sigmoid is as hard as everything else.
  - XorNot 1y ago
    While true, I'd argue that's worse for current progress: the overall trend has been to throw mammoth amounts of electricity at the problem via compute. In so much as something like DeepSeek proves there's gains to be made, the current growth in resource usage would speak against declaring it proves improvement soon. Like plug that issue into known hardware limitations due to physical limits (i.e. we're already up against the wall re: feature sizes) and I'm even more skeptical. If we were looking at a buy down in training resources which was accelerating, then I'd be a lot more interested.

refulgentis 1y ago

Is the slatestarcodex guy "well-versed in all related fields"? Isn't he a psychologist? What would being well versed in all related fields even mean? Especially in the context of the output, a fictional overthetop geopolitics text that leaves the AI stuff at "at timestamp N+1, the model gets better" It's of the same stuff as fan fiction, layers of odd geopolitical stuff, no science fiction. Even at that it is internally incoherent quite regularly (the White House looks to jail the USA Champion AI Guy for some reason while they're in the midst of declaring it an existential war against China) Titillating, in that Important Things are happening. Sophomoric, in that the important things are off camera and an excuse to talk about something else. I say that as someone who believes people 20 years from now will say it happened somewhere between sonnets agentic awareness and o3s uncanny post human ability to turn a factual inquiry about the ending of a TV show into an incisive therapy session

e1g 1y ago
The prime mover behind this project is Daniel Kokotajlo, an ex-OpenAI researcher who documented his last predictions in 2021 [1], and much of that essay turned out to be nearly prophetic. Scott Alexander is a psychiatrist, but more relevant is that he dedicated the last decade to thinking and writing about societal forces, which is useful when forecasting AI. Other contributors are professional AI researchers and forecasters. [1] https://www.lesswrong.com/posts/6Xgy6CAf2jqHhynHL/what-2026-looks-like https://www.lesswrong.com/posts/6Xgy6CAf2jqHhynHL/what-2026-...
- refulgentis 1y ago
  Oh my. I had no idea until now, that was exactly the same flavor and apparently, this is no coincidence. I'm not sure it was prophetic, it was a good survey of the field, but the claim was...a plot of grade schooler to PhD against year. I'm glad he got a paycheck from OpenAI at one point in time. I got one from Google in one point in time. Both of these projects are puffery, not scientific claims of anything, or claims of anything at all other than "at timestamp N+1, AI will be better than timestamp N, on an exponential curve" Utterly bog-standard boring claim going back to 2016 AFAIK. Not the product of considered expertise. Not prophetic.

ajb 1y ago

There's no indication that any of them are well versed in anything to do with the physical world (manufacturing, electronics, agriculture, etc) but they forecast that AI can replace the human physical economy in a few years, including manufacturing it's own chips.

popcorncowboy 1y ago
Because 1, 2, skip a few, 99, ASI. Check, and mate.
lordofgibbons 1y ago
> but they forecast that AI can replace the human physical economy in a few years I guess it depends how many years YOU mean. They're absolutely not claiming that there will be armies of robots making chips in 3 years. They're claiming there will be some semblance of AGI that will be capable of improving/speeding-up the AI development loop within 3 years.
- ajb 1y ago
  The ai-2027 folks absolutely do claim that. Their scenario has "Humans realize that they are obsolete" in late 2029.
- pjc50 1y ago
  > They're absolutely not claiming that there will be armies of robots making chips in 3 years. They're claiming there will be some semblance of AGI that will be capable of improving/speeding-up the AI development loop within 3 years Motte and bailey: huge claim in the headline, little tiny claim in the body.

amarcheschi 1y ago

That's more of a blog article than a research paper... Scott Alexander (one of the writers), yudkowsky, and the others (not the other authors, the other group of "thinkers" with similar ideas) are more or less Ai doomers with no actual background in machine learning/ai I don't think why we should listen to them. Especially when that blog page is formatted in a deceptive way to look like a research paper It's not science, it's science fiction

ben_w 1y ago
> are more or less Ai doomers with no actual background in machine learning/ai I don't think why we should listen to them. Weather vs. climate. The question they're asking isn't about machine learning specifically, it's about the risks of generic optimisers optimising a utility function, and the difficulty of specifying a utility function in a way that doesn't have unfortunate side effects. The examples they give also work with biology (genetics and the difference between what your genes "want" and what your brain "wants") and with governance (laws and loopholes, cobra effects, etc.). This is why a lot (I don't want to say "majority") of people who do have an actual background in machine learning and AI, pay attention to doomer arguments. Some of them* may be business leaders using the same language to BS their way into regulatory capture, but my experience of "real" AI researchers is they're mostly also "safety is important, Yudkowsky makes good points about XYZ" even if they would also say "my P(doom) is only 10%, not 95% like Yudkowsky". * I'm mainly thinking of Musk here, thanks to him saying "AI is summoning the demon" while also having an AI car company, funding OpenAI in the early years and now being in a legal spat with it that looks like it's "hostile takeover or interfere to the same end", funding another AI company, building humanoid robots and showing off ridiculous compute hardware, having brain implant chips, etc.
- amarcheschi 1y ago
  >The question they're asking isn't about machine learning specifically, it's about the risks of generic optimisers optimising a utility function, and the difficulty of specifying a utility function in a way that doesn't have unfortunate side effects. The examples they give also work with biology (genetics and the difference between what your genes "want" and what your brain "wants") and with governance (laws and loopholes, cobra effects, etc.). But you do need some kind of base knowledge, if you want to talk about this. Otherwise you're saying "what if we create God". And last time I checked it wasn't possible. And what's with the existential risk obsession? That's like a bad retelling of the Pascal bet on the existence of God. I'm relieved that at least in italy I still have to find someone in Ai taking them into consideration for more than a few minutes during an ethics course (with students sneering at the ideas of bostrom possible futures), and again, it's held by a professor with no technical knowledge with whom i often disagree due to this