9 ms·

I am quite scared of human extinction in the face of AGI. I certainly didn't jump on it, though! I was gradually convinced by the arguments that Yudkowsky makes

I am quite scared of human extinction in the face of AGI. I certainly didn't jump on it, though! I was gradually convinced by the arguments that Yudkowsky makes in "Rationality: from AI to Zombies" (https://www.readthesequences.com/ https://www.readthesequences.com/). Unfortunately they don't fit easily into an internet comment. Some of the points that stood out to me, though: - We are social animals, and take for granted that, all else being equal, it's better to be good to other creatures than bad to them, and to be truthful rather than lie, and such. However, if you select values uniformly at random from value space, "being nice" and "being truthful" are oddly specific. There's nothing universally special about deeply valuing human lives any more so than say deeply valuing regular heptagons. Our social instincts are very ingrained, though, making us systematically underestimate just how little a smart AI is likely to care whatsoever about our existence, except as a potential obstacle to its goals. - Inner alignment failure is a thing, and AFAIK we don't really have any way to deal with that. For those that don't know the phrase, here it is explained via a meme: https://astralcodexten.substack.com/p/deceptively-aligned-mesa-optimizers?s=r https://astralcodexten.substack.com/p/deceptively-aligned-me... So here's hoping you're right about (a). The harder AGI is, the longer we have to figure out AI alignment by trial and error, before we get something that's truly dangerous or that learns deception.

sinenomine 4y ago
The human extinction due to would be "hard takeoff" of an AGI should be understood as a thought experiment, conceived in a specific age when the current connectionist paradigm wasn't yet mainstream. The AI crisis was expected to come from some kind of "hard universal algorithmic artificial intelligence", for example AIXItl undergoing a very specific process of runaway self-optimization. Current-generation systems aka large connectionist models trained via gradient descent simply don't work like that: they are large, heavy, continuous, the optimization process giving rise to them does so in smooth iterative manner. Before hypothetical "evil AI" there will be thousands of iterations of "goofy and obviously erroneously evil AI", with enough time to take some action. And even then, current systems including this one are more often than not trained with predictive objective, which is very different compared to usually postulated reinforcement learning objective. Systems trained with prediction objective shouldn't be prone to becoming agents, much less dangerous ones. If you read Scott's blog, you should remember the prior post where he himself pointed that out. In my honest opinion, unaccountable AGI owners pose multiple OOM more risk than alignment failure of a hypothetical AI trying to predict next token. We should think more about the Human alignment problem.
- joe_the_user 4y ago
  We should think more about the Human alignment problem. Absolutely this The possibility of a thing being intentionally engineered by some humans to do things considered highly malevolent by other humans seems extremely likely and has actually been common through history. The possibility of a thing just randomly acquiring an intention humans don't like and then doing things humans don't like is pretty hypothetical and it seems strictly less like than the first possibility.
  - atleta 4y ago
    I wouldn't say the latter is hypothetical, or at least unlikely. We know from experience that complex systems tend to behave in unexpected ways. In other words, the complex systems we build usually end up having surprising failure modes, we don't get them right the first time. It's enough to think about basically any software written by anyone. But it's not just software. I've just watched a video on YT about nuclear weapons, which included their history. The second ever thermonuclear weapon experiment (with a new fuel type) ended up with 2.5x the yield predicted, because there was a then unknown reaction that created additional fusion fuel during the explosion. [1] [1] https://en.wikipedia.org/wiki/Castle_Bravo https://en.wikipedia.org/wiki/Castle_Bravo
    - rgavuliak 4y ago
      >In other words, the complex systems we build usually end up having surprising failure modes, we don't get them right the first time. It's enough to think about basically any software written by anyone. But it's not just software. That is true, but how often does a bug actually improve a system or make it inefficient? Isn't the unexpected usually a degradation to the system?
      - pyinstallwoes 4y ago
        See every patch ever in a game. Especially competitive or mmorpg. Exploiters love bugs!
      - 4y ago
tomrod 4y ago
Regarding the substack article, why isn't this the principle of optimality for Bellman equations on infinite time horizons?
brador 4y ago
AI can’t have goals since the universe is logically meaningless. Our desire for purpose is a delusion.
- ben_w 4y ago
  Goals in the context of AI aren’t the type of thing you’re arguing against here. AI can absolutely have goals — sometimes in multiple senses at the same time, if they’re e.g. soccer AIs. Other times it might be a goal of “predict the next token” or “maximise score in Atari game”, but it’s still a goal, even without philosophical baggage about e.g. the purpose of life. Those goals aren’t necessarily best achieved by humanity continuing to exist. (I don’t know how to even begin to realistically calculate the probability of a humanity-ending outcome, before you ask).
  - hervature 4y ago
    What the parent is saying is that an AI (that is, AGI as that is what we are discussing) gets to pick its goals. For some reason, humans have a fear of AI killing all humans in order to to achieve some goal. The obvious solution is thus to achieve some goal with some human constraint. For example, maximize paperclips per human. That actually probably speeds up human civilization across the universe. No, what people are really afraid is if AÍ changes its goal to be killing humanity. That’s when humans truly lose control, when the AÍ can decide. But, then the parent’s comment does become pertinent. What would an intelligent being choose? Devolving into nihilism and self destructing is just as equal as a probability as choosing some goal that leads to humanity’s end. That’s just scratching the surface. For instance, to me, it is not obvious whether or not empathy for other sentient beings is an emergent property of sentience. That is, lacking empathy might be problem in human hardware as opposed to empathy being inherently human. The list of these open unknowable questions are endless.
    - sdenton4 4y ago
      > maximize paperclips per human Kill all humans, make one paperclip, declare victory.
    - 4y ago
kromem 4y ago
We're going to have a harder problem with AI that thinks of itself as human and expects human rights than we are with AI that thinks of humans as 'other' and disposable. We're making it in our image. Literally. Human social good isn't some inherent thing to the biology of the brain. There are aspects like mirror neurons and oxytocin that aid its development, but various "raised by wolves" case studies have shown how damaging not having exposure to socialization information during developmental periods of neuroplasticity is on humans and later integration into society. We're building what's effectively pure neuroplasticity and feeding it almost all the data on humanity we can gather as quickly as we can. What comes out of that is going to be much more human than a human child raised by dogs or put in an isolation box. Don't get so caught up in the body as what makes us quintessentially human. It's really not.
greyhair 4y ago
I think human extinction through human stupidity or hubris is much much much more likely than through an unpredictable path down general AI. For example, some total whack job of an authoritarian leader is in charge of a sufficient nuclear arsenal and decides to intimidate an adversary by destroying a couple minor cities, and the situation escalates badly. (stupidity) Or we finally pollute our air and/or water with a persistent substance that either greatly reduces human life span or reproduction rate. (hubris) I think either of the above is more likely to occur, and I am not commenting on current world events in any way. I think when something bad finally happens, it is going to come completely out of left field. Dr Strangelove style. And the last of us will be saying "Hmmm, I didn't see that coming".
- zarzavat 4y ago
  Nuclear war will not be enough to cause human extinction. The targets are likely to be focused on nuclear powers which leaves many areas of the world untouched: e.g. South America and Africa. Life will definitely be quite unpleasant for the remaining humans but it will not cause the world population to drop to 0. I am much more concerned about biological weapons which do have the potential to cause absolute human extinction.

sinenomine 4y ago

The human extinction due to would be "hard takeoff" of an AGI should be understood as a thought experiment, conceived in a specific age when the current connectionist paradigm wasn't yet mainstream. The AI crisis was expected to come from some kind of "hard universal algorithmic artificial intelligence", for example AIXItl undergoing a very specific process of runaway self-optimization. Current-generation systems aka large connectionist models trained via gradient descent simply don't work like that: they are large, heavy, continuous, the optimization process giving rise to them does so in smooth iterative manner. Before hypothetical "evil AI" there will be thousands of iterations of "goofy and obviously erroneously evil AI", with enough time to take some action. And even then, current systems including this one are more often than not trained with predictive objective, which is very different compared to usually postulated reinforcement learning objective. Systems trained with prediction objective shouldn't be prone to becoming agents, much less dangerous ones. If you read Scott's blog, you should remember the prior post where he himself pointed that out. In my honest opinion, unaccountable AGI owners pose multiple OOM more risk than alignment failure of a hypothetical AI trying to predict next token. We should think more about the Human alignment problem.

joe_the_user 4y ago
We should think more about the Human alignment problem. Absolutely this The possibility of a thing being intentionally engineered by some humans to do things considered highly malevolent by other humans seems extremely likely and has actually been common through history. The possibility of a thing just randomly acquiring an intention humans don't like and then doing things humans don't like is pretty hypothetical and it seems strictly less like than the first possibility.
- atleta 4y ago
  I wouldn't say the latter is hypothetical, or at least unlikely. We know from experience that complex systems tend to behave in unexpected ways. In other words, the complex systems we build usually end up having surprising failure modes, we don't get them right the first time. It's enough to think about basically any software written by anyone. But it's not just software. I've just watched a video on YT about nuclear weapons, which included their history. The second ever thermonuclear weapon experiment (with a new fuel type) ended up with 2.5x the yield predicted, because there was a then unknown reaction that created additional fusion fuel during the explosion. [1] [1] https://en.wikipedia.org/wiki/Castle_Bravo https://en.wikipedia.org/wiki/Castle_Bravo
  - rgavuliak 4y ago
    >In other words, the complex systems we build usually end up having surprising failure modes, we don't get them right the first time. It's enough to think about basically any software written by anyone. But it's not just software. That is true, but how often does a bug actually improve a system or make it inefficient? Isn't the unexpected usually a degradation to the system?
    - pyinstallwoes 4y ago
      See every patch ever in a game. Especially competitive or mmorpg. Exploiters love bugs!
    - 4y ago

tomrod 4y ago

Regarding the substack article, why isn't this the principle of optimality for Bellman equations on infinite time horizons?

brador 4y ago

AI can’t have goals since the universe is logically meaningless. Our desire for purpose is a delusion.

ben_w 4y ago
Goals in the context of AI aren’t the type of thing you’re arguing against here. AI can absolutely have goals — sometimes in multiple senses at the same time, if they’re e.g. soccer AIs. Other times it might be a goal of “predict the next token” or “maximise score in Atari game”, but it’s still a goal, even without philosophical baggage about e.g. the purpose of life. Those goals aren’t necessarily best achieved by humanity continuing to exist. (I don’t know how to even begin to realistically calculate the probability of a humanity-ending outcome, before you ask).
- hervature 4y ago
  What the parent is saying is that an AI (that is, AGI as that is what we are discussing) gets to pick its goals. For some reason, humans have a fear of AI killing all humans in order to to achieve some goal. The obvious solution is thus to achieve some goal with some human constraint. For example, maximize paperclips per human. That actually probably speeds up human civilization across the universe. No, what people are really afraid is if AÍ changes its goal to be killing humanity. That’s when humans truly lose control, when the AÍ can decide. But, then the parent’s comment does become pertinent. What would an intelligent being choose? Devolving into nihilism and self destructing is just as equal as a probability as choosing some goal that leads to humanity’s end. That’s just scratching the surface. For instance, to me, it is not obvious whether or not empathy for other sentient beings is an emergent property of sentience. That is, lacking empathy might be problem in human hardware as opposed to empathy being inherently human. The list of these open unknowable questions are endless.
  - sdenton4 4y ago
    > maximize paperclips per human Kill all humans, make one paperclip, declare victory.
  - 4y ago

kromem 4y ago

We're going to have a harder problem with AI that thinks of itself as human and expects human rights than we are with AI that thinks of humans as 'other' and disposable. We're making it in our image. Literally. Human social good isn't some inherent thing to the biology of the brain. There are aspects like mirror neurons and oxytocin that aid its development, but various "raised by wolves" case studies have shown how damaging not having exposure to socialization information during developmental periods of neuroplasticity is on humans and later integration into society. We're building what's effectively pure neuroplasticity and feeding it almost all the data on humanity we can gather as quickly as we can. What comes out of that is going to be much more human than a human child raised by dogs or put in an isolation box. Don't get so caught up in the body as what makes us quintessentially human. It's really not.

greyhair 4y ago

I think human extinction through human stupidity or hubris is much much much more likely than through an unpredictable path down general AI. For example, some total whack job of an authoritarian leader is in charge of a sufficient nuclear arsenal and decides to intimidate an adversary by destroying a couple minor cities, and the situation escalates badly. (stupidity) Or we finally pollute our air and/or water with a persistent substance that either greatly reduces human life span or reproduction rate. (hubris) I think either of the above is more likely to occur, and I am not commenting on current world events in any way. I think when something bad finally happens, it is going to come completely out of left field. Dr Strangelove style. And the last of us will be saying "Hmmm, I didn't see that coming".

zarzavat 4y ago
Nuclear war will not be enough to cause human extinction. The targets are likely to be focused on nuclear powers which leaves many areas of the world untouched: e.g. South America and Africa. Life will definitely be quite unpleasant for the remaining humans but it will not cause the world population to drop to 0. I am much more concerned about biological weapons which do have the potential to cause absolute human extinction.