6 ms·

Transformative AGI by 2043 is <1% likely

33 points by snewman 3y ago

deleted 3y ago
[deleted]
humanistbot 3y ago
So this is some kind of attempt to make a Drake Equation [1] for AGI? That's more useful as a thought experiment than something claimed with scientific precision. [1] https://en.wikipedia.org/wiki/Drake_equation https://en.wikipedia.org/wiki/Drake_equation
- tuatoru 3y ago
  Any atttept to apply skeptical (in the proper sense of the word) thought to AI is valuable.
celdon25 3y ago
Using the tool referenced in the paper, my own estimate came to around 25% https://www.tedsanders.com/agi-forecaster/ https://www.tedsanders.com/agi-forecaster/
- CSMastermind 3y ago
  Mine came out to 0%, which is also personally where I'd put it. But I think it's a question wrongly put. We don't need AGI for current tools and their descendants to be extremely disruptive. For the longest time chess programs only mimicked one aspect of the way humans play the game (calculation) but they did it at such a superhuman level we were unable to beat them. LLMs, computer vision tools, etc. don't need to be AGI to be able to displace large amounts of human jobs. Having the discussion about if and in what ways we can prepare for the innovation that seems inevitable in that space seems like a much more prudent discussion than speculating about whether AGI will exist or not.
  - drdeca 3y ago
    “0 is not a probability”
eldritch_4ier 3y ago
Don’t we all known someone who is very widely read, but makes the occasional error or mistake? Gets confused about something, or is asked about a discipline they don’t understand or haven’t researched very deeply? Maybe they misremember a date here or there, but are otherwise fairly intelligent? Maybe they work a data entry job making a middle class salary. This is basically where ChatGPT is at. It’s a very widely read person with an excellent memory and quick mind. It’s probably smarter than the average human (certainly in breadth, often in depth), not to mention a comparison of it to the average human globally. ChatGPT is smarter than the average human already. It doesn’t need agency or a soul to do so. We already have AGI, we just keep moving the goal posts.
- adastra22 3y ago
  We’ve had AGI since 2017. I find it very hard to take AI predictions seriously unless this fact is acknowledged.
  - jononor 3y ago
    Which AGI are you referring to?
    - lustysocietyorg 3y ago
      [dead]
    - gorbypark 3y ago
      Just a guess as I'm not OP, but 2017 is when the "Attention Is All You Need" paper was published.
    - adastra22 3y ago
      Attention-based transformer architectures of the same type as GPT were first published in 2017.
      - dragonwriter 3y ago
anonuser123456 3y ago
“we estimate that transformative AGI by 2043 is 0.4% likely” Not enough significant digits to take seriously. Now if they had said 0.39785% they might have some credibility.
adverbly 3y ago
Title isn't quite right... The article is estimating the probability of widespread/consumer accessible/cost effective AGI. But AGI can be transformative without being widespread. Replacing all human functions is not nessecary for transformation. A single superhumam AI could cost 1e6 that of a single human, but still do transformative work if it's 1.5x smarter than us. Edit: nevermind... This was an intentional definition of "transformative" made by the authors. Seems wrong to me, but they were up front about it. Not impressed though. Anyone can drum attention by using a non-standard definition of something. But of a cheap trick imo, but that's academia these days I guess... For reference, my definition of transformative would be "responsible for large scale changes to society". In other words, it doesn't need to happen at scale for impact to happen at scale. Replication of a solution is cheap once it is discovered.
- carlmr 3y ago
  I think if we get 1.5x as smart as the smartest human, then that will automatically scale after a few iterations of improving itself.
  - kibibu 3y ago
    There should be a name for this idea. I propose The Kurzweilarity.
    - headcanon 3y ago
      There already is a term for it: "Foom" https://en.wiktionary.org/wiki/foom https://en.wiktionary.org/wiki/foom
  - TrapLord_Rhodo 3y ago
    "As Smart" I don't think it needs that at all. It just needs to 'Code' 1.01 times better than the average programmer to enter a self improvement cycle.
- pama 3y ago
  It appears like a very poor analysis overall. These closing arguments suggest lack of deep thinking: “Consider Alphabet, the parent company of Google DeepMind. Alphabet is not constrained by financial capital: as of March 2023, Alphabet had $115B of cash on hand.202 Alphabet is also not constrained by human capital: as of March 2023, Alphabet had 191,000 employees.203 If Alphabet wanted to, it could easily hire more AI researchers and allocate more software engineers to AI research infrastructure. But it chooses not to. Why? This deliberate choice presumably implies that Alphabet management doesn’t believe there is enough ROI from hiring more smart people to accelerate progress on AGI research. To us, this suggests that Alphabet management is bearish on the likelihood of transformative AGI by 2043. We also see this as a stronger signal than public markets, as Alphabet management is likely much more informed about AGI progress than the average stock trader. Another implicit signal of low likelihood of transformative AGI is the comparatively low valuation of AGI companies. Consider Anthropic, which reportedly raised money in early 2023 at a valuation of ~$4B. $4B is ~0.004% of world product each year. If transformative AGI is soon able to do most human work, we expect it to be worth multiples of today’s world product. Supposing transformative AGI is worth $100T/yr, and Anthropic has a 10% chance of capturing 10% of the value for an expected duration of 10 years, that would imply a valuation of $10T (before time discounting). $4B is less than 1/1,000th of $10T, which seems to imply that Anthropic and its investors do not collectively believe that Anthropic has a chance of inventing transformative AGI by 2043 and capturing its value.”
ajuc 3y ago
Let's say we get AGI that costs billions to run, can't do some simple things well, don't have good cheap robots to use, but discovers a way to lengthen the human life significantly and cure most illnesses. Pretty transformative if you ask me.
- johnnyworker 3y ago
  > don't have good cheap robots to use https://pluralistic.net/2021/02/17/reverse-centaur/#reverse-centaur https://pluralistic.net/2021/02/17/reverse-centaur/#reverse-... > Pretty transformative if you ask me. It's a fairy tale that only serves to play for time. Fewer and fewer control more and more, and they will not suddenly play nice once they no longer need the consent of workers and/or the public, or can force people without their consent. That never passed the smell test. Life, as a whole, already is immortal. Immortal individuals will at best kick away the ladder, and stop the stream of life. So they can live forever, and nobody else gets born. Only the people who are the worst at living, and the least wise, would even want this. I noticed this as early as the late 90s... the people who were excited about immortality exuded all sorts of things, but decidedly not life. That hasn't changed. It's like power, those who want it the most are the ones who should under no circumstances be allowed to have it.
- lm28469 3y ago
  Most death are lifestyle related, you already can extend your life significantly. Also it would finish hour already crumbling system
albertTJames 3y ago
This must be one of the stupidest article I have ever read. The first page only shows they have absolutely no clue how statistics works. Of course if you consider every events in the universe completely independent the probability of any large combination of those will be very low. But it is of course not the case with all the conditions they describe, those are highly correlated and P(A|B)P(B)>P(A)P(B) for all of those.
- inimino 3y ago
  > For each step, we estimate a probability of success by 2043, conditional on prior steps being achieved. [emphasis added] Maybe your reading comprehension is the problem.
  - albertTJames 3y ago
    You are absolutely right. AGI is already here anyways.
RetroTechie 3y ago
Impossible to predict future if future depends on unpredictable factors. I can't shake the feeling that recent developments like LLM's, ChatGPT & co are barking up the wrong tree. Or are missing a key piece of the puzzle. Or that vastly simpler (computationally cheaper) constructs with similar capabilities could be found. That in hindsight (say, 20y from now) we'll say "see, it was really easy!". That missing piece(s) were small but essential. And (perhaps) non-obvious right now. See https://longbets.org/1/ https://longbets.org/1/ Ray Kurzweil's argument is well reasoned & very convincing imho. Computing power will get there or already is. And our understanding of the architecture & function of the human brain is a steadily-completing picture. Not to mention brains of smaller creatures. All it takes is time. Doubtful about timeframe in above bet. And efficiencies of artificial vs. biological brains remains to be seen. But yes, AGI will be achieved. Likely sooner than later.
- willow8349 3y ago
  Look at the power draw of the human brains versus any of the hardware running these models. It’s an order of magnitude difference. Often multiple orders of magnitude. Part of the reason is a massive focus on GPUs and digital techniques. Some things like math operations are quite fast and accurate to do in a discrete sense when that matters. The human brain is not discrete though. Nor is it really fully analog. Specialized hardware beyond GPUs for parallel digital calculations will likely be needed to implement architectures that get actually closer to human levels of intelligence.
  - davidktr 3y ago
    Deep down the brain is 100% discrete: Neurons are either firing or not. To me, the brain's biggest mistery is how it goes from this to doing all the analogue stuff, and ends up with our capacity to deal with symbols.
    - montereynack 3y ago
      Sorry, but this is untrue; action potentials in neurons have extremely complex interplay with each other, including residual “soft” periods and chemically-induced changes in how they fire. Neurons don’t just “fire” or “not fire”, they adaptively change the strength of their firing constantly, unpredictably and continuously.
manuhortet 3y ago
"China has stated plainly it intends to reunify (invade) Taiwan. A majority of its population supports invasion. Its military is preparing to be ready for invasion" Am I reading a paper or listening to Rogan?
hoseja 3y ago
>look into paper >it's Drake equation
jospf 3y ago
The only consistency in AI predictions is to be wrong, whether it's 'AI will never be able to do....' or GAI by year 20xx.
mark_l_watson 3y ago
My opinion on this may be worthless since I have worked in AI since 1982, and I am guilty of enjoying living through several AI hype cycles. After seeing surprising advances in techniques for deep models, then attention+transformer models, I think there is a lot of progress still to be made with cooperating LLMs. I have a simple example of this in the last book I wrote. I have no idea what new ideas will work, but I would be very surprised if every 4 or 5 years new and fresh ideas don’t occur to achieve really good reasoning, counterfactual reasoning, etc. Anyway, putting the chance of transformative AGI below 1% for the next 20 years seems wrong to me.
cubefox 3y ago
The standard response to this is that the above is commiting the multiple stages fallacy: https://arbital.com/p/multiple_stage_fallacy/ https://arbital.com/p/multiple_stage_fallacy/
jimrandomh 3y ago
Previous discussion on LessWrong: https://www.lesswrong.com/posts/DgzdLzDGsqoRXhCK7/transformative-agi-by-2043-is-less-than-1-likely https://www.lesswrong.com/posts/DgzdLzDGsqoRXhCK7/transforma... and EA Forum: https://forum.effectivealtruism.org/posts/ARkbWch5RMsj6xP5p/transformative-agi-by-2043-is-less-than-1-likely https://forum.effectivealtruism.org/posts/ARkbWch5RMsj6xP5p/... Both previous discussions contain multiple independent refutations of the core claim (the argument stacks quite a few errors on top of each other).

33 points by snewman 3y ago

deleted 3y ago

[deleted]

So this is some kind of attempt to make a Drake Equation [1] for AGI? That's more useful as a thought experiment than something claimed with scientific precision. [1] https://en.wikipedia.org/wiki/Drake_equation https://en.wikipedia.org/wiki/Drake_equation

tuatoru 3y ago
Any atttept to apply skeptical (in the proper sense of the word) thought to AI is valuable.

celdon25 3y ago

Using the tool referenced in the paper, my own estimate came to around 25% https://www.tedsanders.com/agi-forecaster/ https://www.tedsanders.com/agi-forecaster/

CSMastermind 3y ago
Mine came out to 0%, which is also personally where I'd put it. But I think it's a question wrongly put. We don't need AGI for current tools and their descendants to be extremely disruptive. For the longest time chess programs only mimicked one aspect of the way humans play the game (calculation) but they did it at such a superhuman level we were unable to beat them. LLMs, computer vision tools, etc. don't need to be AGI to be able to displace large amounts of human jobs. Having the discussion about if and in what ways we can prepare for the innovation that seems inevitable in that space seems like a much more prudent discussion than speculating about whether AGI will exist or not.
- drdeca 3y ago
  “0 is not a probability”

eldritch_4ier 3y ago

Don’t we all known someone who is very widely read, but makes the occasional error or mistake? Gets confused about something, or is asked about a discipline they don’t understand or haven’t researched very deeply? Maybe they misremember a date here or there, but are otherwise fairly intelligent? Maybe they work a data entry job making a middle class salary. This is basically where ChatGPT is at. It’s a very widely read person with an excellent memory and quick mind. It’s probably smarter than the average human (certainly in breadth, often in depth), not to mention a comparison of it to the average human globally. ChatGPT is smarter than the average human already. It doesn’t need agency or a soul to do so. We already have AGI, we just keep moving the goal posts.

adastra22 3y ago
We’ve had AGI since 2017. I find it very hard to take AI predictions seriously unless this fact is acknowledged.
- jononor 3y ago
  Which AGI are you referring to?
  - lustysocietyorg 3y ago
    [dead]
  - gorbypark 3y ago
    Just a guess as I'm not OP, but 2017 is when the "Attention Is All You Need" paper was published.
  - adastra22 3y ago
    Attention-based transformer architectures of the same type as GPT were first published in 2017.
    - dragonwriter 3y ago

anonuser123456 3y ago

“we estimate that transformative AGI by 2043 is 0.4% likely” Not enough significant digits to take seriously. Now if they had said 0.39785% they might have some credibility.

adverbly 3y ago

Title isn't quite right... The article is estimating the probability of widespread/consumer accessible/cost effective AGI. But AGI can be transformative without being widespread. Replacing all human functions is not nessecary for transformation. A single superhumam AI could cost 1e6 that of a single human, but still do transformative work if it's 1.5x smarter than us. Edit: nevermind... This was an intentional definition of "transformative" made by the authors. Seems wrong to me, but they were up front about it. Not impressed though. Anyone can drum attention by using a non-standard definition of something. But of a cheap trick imo, but that's academia these days I guess... For reference, my definition of transformative would be "responsible for large scale changes to society". In other words, it doesn't need to happen at scale for impact to happen at scale. Replication of a solution is cheap once it is discovered.

carlmr 3y ago
I think if we get 1.5x as smart as the smartest human, then that will automatically scale after a few iterations of improving itself.
- kibibu 3y ago
  There should be a name for this idea. I propose The Kurzweilarity.
  - headcanon 3y ago
    There already is a term for it: "Foom" https://en.wiktionary.org/wiki/foom https://en.wiktionary.org/wiki/foom
- TrapLord_Rhodo 3y ago
  "As Smart" I don't think it needs that at all. It just needs to 'Code' 1.01 times better than the average programmer to enter a self improvement cycle.
pama 3y ago
It appears like a very poor analysis overall. These closing arguments suggest lack of deep thinking: “Consider Alphabet, the parent company of Google DeepMind. Alphabet is not constrained by financial capital: as of March 2023, Alphabet had $115B of cash on hand.202 Alphabet is also not constrained by human capital: as of March 2023, Alphabet had 191,000 employees.203 If Alphabet wanted to, it could easily hire more AI researchers and allocate more software engineers to AI research infrastructure. But it chooses not to. Why? This deliberate choice presumably implies that Alphabet management doesn’t believe there is enough ROI from hiring more smart people to accelerate progress on AGI research. To us, this suggests that Alphabet management is bearish on the likelihood of transformative AGI by 2043. We also see this as a stronger signal than public markets, as Alphabet management is likely much more informed about AGI progress than the average stock trader. Another implicit signal of low likelihood of transformative AGI is the comparatively low valuation of AGI companies. Consider Anthropic, which reportedly raised money in early 2023 at a valuation of ~$4B. $4B is ~0.004% of world product each year. If transformative AGI is soon able to do most human work, we expect it to be worth multiples of today’s world product. Supposing transformative AGI is worth $100T/yr, and Anthropic has a 10% chance of capturing 10% of the value for an expected duration of 10 years, that would imply a valuation of $10T (before time discounting). $4B is less than 1/1,000th of $10T, which seems to imply that Anthropic and its investors do not collectively believe that Anthropic has a chance of inventing transformative AGI by 2043 and capturing its value.”

ajuc 3y ago

Let's say we get AGI that costs billions to run, can't do some simple things well, don't have good cheap robots to use, but discovers a way to lengthen the human life significantly and cure most illnesses. Pretty transformative if you ask me.

johnnyworker 3y ago
> don't have good cheap robots to use https://pluralistic.net/2021/02/17/reverse-centaur/#reverse-centaur https://pluralistic.net/2021/02/17/reverse-centaur/#reverse-... > Pretty transformative if you ask me. It's a fairy tale that only serves to play for time. Fewer and fewer control more and more, and they will not suddenly play nice once they no longer need the consent of workers and/or the public, or can force people without their consent. That never passed the smell test. Life, as a whole, already is immortal. Immortal individuals will at best kick away the ladder, and stop the stream of life. So they can live forever, and nobody else gets born. Only the people who are the worst at living, and the least wise, would even want this. I noticed this as early as the late 90s... the people who were excited about immortality exuded all sorts of things, but decidedly not life. That hasn't changed. It's like power, those who want it the most are the ones who should under no circumstances be allowed to have it.
lm28469 3y ago
Most death are lifestyle related, you already can extend your life significantly. Also it would finish hour already crumbling system

albertTJames 3y ago

This must be one of the stupidest article I have ever read. The first page only shows they have absolutely no clue how statistics works. Of course if you consider every events in the universe completely independent the probability of any large combination of those will be very low. But it is of course not the case with all the conditions they describe, those are highly correlated and P(A|B)P(B)>P(A)P(B) for all of those.

inimino 3y ago
> For each step, we estimate a probability of success by 2043, conditional on prior steps being achieved. [emphasis added] Maybe your reading comprehension is the problem.
- albertTJames 3y ago
  You are absolutely right. AGI is already here anyways.

RetroTechie 3y ago

Impossible to predict future if future depends on unpredictable factors. I can't shake the feeling that recent developments like LLM's, ChatGPT & co are barking up the wrong tree. Or are missing a key piece of the puzzle. Or that vastly simpler (computationally cheaper) constructs with similar capabilities could be found. That in hindsight (say, 20y from now) we'll say "see, it was really easy!". That missing piece(s) were small but essential. And (perhaps) non-obvious right now. See https://longbets.org/1/ https://longbets.org/1/ Ray Kurzweil's argument is well reasoned & very convincing imho. Computing power will get there or already is. And our understanding of the architecture & function of the human brain is a steadily-completing picture. Not to mention brains of smaller creatures. All it takes is time. Doubtful about timeframe in above bet. And efficiencies of artificial vs. biological brains remains to be seen. But yes, AGI will be achieved. Likely sooner than later.

willow8349 3y ago
Look at the power draw of the human brains versus any of the hardware running these models. It’s an order of magnitude difference. Often multiple orders of magnitude. Part of the reason is a massive focus on GPUs and digital techniques. Some things like math operations are quite fast and accurate to do in a discrete sense when that matters. The human brain is not discrete though. Nor is it really fully analog. Specialized hardware beyond GPUs for parallel digital calculations will likely be needed to implement architectures that get actually closer to human levels of intelligence.
- davidktr 3y ago
  Deep down the brain is 100% discrete: Neurons are either firing or not. To me, the brain's biggest mistery is how it goes from this to doing all the analogue stuff, and ends up with our capacity to deal with symbols.
  - montereynack 3y ago
    Sorry, but this is untrue; action potentials in neurons have extremely complex interplay with each other, including residual “soft” periods and chemically-induced changes in how they fire. Neurons don’t just “fire” or “not fire”, they adaptively change the strength of their firing constantly, unpredictably and continuously.

manuhortet 3y ago

"China has stated plainly it intends to reunify (invade) Taiwan. A majority of its population supports invasion. Its military is preparing to be ready for invasion" Am I reading a paper or listening to Rogan?

hoseja 3y ago

>look into paper >it's Drake equation

jospf 3y ago

The only consistency in AI predictions is to be wrong, whether it's 'AI will never be able to do....' or GAI by year 20xx.

mark_l_watson 3y ago

My opinion on this may be worthless since I have worked in AI since 1982, and I am guilty of enjoying living through several AI hype cycles. After seeing surprising advances in techniques for deep models, then attention+transformer models, I think there is a lot of progress still to be made with cooperating LLMs. I have a simple example of this in the last book I wrote. I have no idea what new ideas will work, but I would be very surprised if every 4 or 5 years new and fresh ideas don’t occur to achieve really good reasoning, counterfactual reasoning, etc. Anyway, putting the chance of transformative AGI below 1% for the next 20 years seems wrong to me.

cubefox 3y ago

The standard response to this is that the above is commiting the multiple stages fallacy: https://arbital.com/p/multiple_stage_fallacy/ https://arbital.com/p/multiple_stage_fallacy/

jimrandomh 3y ago

Previous discussion on LessWrong: https://www.lesswrong.com/posts/DgzdLzDGsqoRXhCK7/transformative-agi-by-2043-is-less-than-1-likely https://www.lesswrong.com/posts/DgzdLzDGsqoRXhCK7/transforma... and EA Forum: https://forum.effectivealtruism.org/posts/ARkbWch5RMsj6xP5p/transformative-agi-by-2043-is-less-than-1-likely https://forum.effectivealtruism.org/posts/ARkbWch5RMsj6xP5p/... Both previous discussions contain multiple independent refutations of the core claim (the argument stacks quite a few errors on top of each other).