3 ms·

Why do we fear other intelligences anyway? Isn't that just a sign of our own immaturity? Maybe we need to evolve more before we start think about creating AGI..

Why do we fear other intelligences anyway? Isn't that just a sign of our own immaturity? Maybe we need to evolve more before we start think about creating AGI...

edanm 9y ago
Mostly because other intelligences might have more ability than us to achieve their goals, and have different goals. "Intelligence does not imply goals" is the thing to keep in mind here. E.g. what if more intelligent aliens truly believed that the only purpose in the world is proving more mathematical theorems? And decided to turn all of the planet into a giant math-proving machine? Destroying all the planet, all the animals, all the humans, all the art, whatever, all to prove more maths? I love maths, but I'd consider that a pretty bad outcome. And there's no reason that I've ever seen to think that more intelligence implies anything about goals.
- icebraining 9y ago
  We currently have thousands, or even millions of intelligent entities (humans) which think pretty wild and dangerous things. We usually just tell them "shut up Bob and take your meds" and that's it. Sometimes we regrettably kill them. Why would such AGI have the means of turning all of the planet into anything? I mean, sure, I also think the Terminator is a decent movie, but that doesn't make it a reasonable blueprint of the future.
  - musage 9y ago
    > We usually just tell them "shut up Bob and take your meds" and that's it. Other times, Las Vegas and we shut up because we come up empty -- at best talking about gun control as if someone choking 3 people on a bus was substantially better and not still something where we come up empty. > Why would such AGI have the means of turning all of the planet into anything? How does getting off track with that address the main point, namely "Intelligence does not imply goals"? Are you going to prove that while that may be true, there just can't be a way for that to ever have a bad outcome because "Bob"? > I also think the Terminator is a decent movie, but that doesn't make it a reasonable blueprint of the future. Neither is "Bob".
    - icebraining 9y ago
      It wasn't an offtrack; I was pointing out that we already have intelligent beings that don't share our goals. Sometimes it does have bad outcomes, but I don't remember it ever having "destroying all the planet, all the animals, all the humans, all the art". Which is why I'm asking why would adding another intelligent entity make us fear it.
  - emiliobumachar 9y ago
    If human-level intelligence in the upper bound of intelligence, then there's nothing to fear. There's also not much to fear from slightly-smarter-than humans machines. But why would you think that's the case? Because we're the best evolution has come up with so far, here on Earth? "Why would such AGI have the means of turning all of the planet into anything?" An AI as much smarter than a human than a human is smarter than a dog would think of a strategy far, far smarter that what you, or I, or human civilization could come up with.
  - Smaug123 9y ago
    "Pretty wild and dangerous things" - I think you're suffering from a failure of imagination there. Examples of humans who thought something wild and dangerous were Hitler, or Jesus or Mohammed; those people changed the face of the world because they thought something. And those people were human, constrained by evolution to think in certain ways. An AGI doesn't necessarily have to think inside the human box. There is a lot more of "dangerous thought-space" out there than you seem to be accounting for. Your second paragraph: presumably you think it's literally impossible to invent nanotechnology to make yourself omnipotent, but I'm not willing to place that at less likely than 5%. Something clever enough could probably just manipulate the existing social structures to kill all the humans (something very intelligent that truly wanted nuclear war could probably make it happen). Just a little bit of tweaking to smallpox might be able to destroy humanity anyway. Even just the right new religion might do it (witness the great purges by pretty much every religion ever). None of these suggestions are intended as "this is a plausible way of killing those inconvenient humans"; but if each of my suggestions (nanotech, nuclear war, pandemic, grand new idea like a religion) has a five percent chance of working, that's already a 18.5% chance that exactly one of them would work. Once the humans are dead, of course, it has all the time in the world to achieve all its goals.
namelost 9y ago
https://en.wikipedia.org/wiki/Holocene_extinction https://en.wikipedia.org/wiki/Holocene_extinction
lgas 9y ago
Well, most people assume the development of AGI will in short order lead to the rise of superintelligent AGI and then the reason is precedent -- look at how we treat creatures that are less intelligent than we are. If you're lucky you're a dog or a cat and you're kept as a pet and treated as a member of the family, but you're still a pet. If you're not so lucky you're a cow or a pig and used primarily as a food source. If you're even less lucky you're an insect that gets stepped on for wandering too close to the picnic where we're eating the cow and pig while we watch our dog play.
JabavuAdams 9y ago
The Dodos should have feared us. We don't even hate Dodos, but we killed them all anyway.

Why do we fear other intelligences anyway? Isn't that just a sign of our own immaturity? Maybe we need to evolve more before we start think about creating AGI...

edanm 9y ago

Mostly because other intelligences might have more ability than us to achieve their goals, and have different goals. "Intelligence does not imply goals" is the thing to keep in mind here. E.g. what if more intelligent aliens truly believed that the only purpose in the world is proving more mathematical theorems? And decided to turn all of the planet into a giant math-proving machine? Destroying all the planet, all the animals, all the humans, all the art, whatever, all to prove more maths? I love maths, but I'd consider that a pretty bad outcome. And there's no reason that I've ever seen to think that more intelligence implies anything about goals.

icebraining 9y ago
We currently have thousands, or even millions of intelligent entities (humans) which think pretty wild and dangerous things. We usually just tell them "shut up Bob and take your meds" and that's it. Sometimes we regrettably kill them. Why would such AGI have the means of turning all of the planet into anything? I mean, sure, I also think the Terminator is a decent movie, but that doesn't make it a reasonable blueprint of the future.
- musage 9y ago
  > We usually just tell them "shut up Bob and take your meds" and that's it. Other times, Las Vegas and we shut up because we come up empty -- at best talking about gun control as if someone choking 3 people on a bus was substantially better and not still something where we come up empty. > Why would such AGI have the means of turning all of the planet into anything? How does getting off track with that address the main point, namely "Intelligence does not imply goals"? Are you going to prove that while that may be true, there just can't be a way for that to ever have a bad outcome because "Bob"? > I also think the Terminator is a decent movie, but that doesn't make it a reasonable blueprint of the future. Neither is "Bob".
  - icebraining 9y ago
    It wasn't an offtrack; I was pointing out that we already have intelligent beings that don't share our goals. Sometimes it does have bad outcomes, but I don't remember it ever having "destroying all the planet, all the animals, all the humans, all the art". Which is why I'm asking why would adding another intelligent entity make us fear it.
- emiliobumachar 9y ago
  If human-level intelligence in the upper bound of intelligence, then there's nothing to fear. There's also not much to fear from slightly-smarter-than humans machines. But why would you think that's the case? Because we're the best evolution has come up with so far, here on Earth? "Why would such AGI have the means of turning all of the planet into anything?" An AI as much smarter than a human than a human is smarter than a dog would think of a strategy far, far smarter that what you, or I, or human civilization could come up with.
- Smaug123 9y ago
  "Pretty wild and dangerous things" - I think you're suffering from a failure of imagination there. Examples of humans who thought something wild and dangerous were Hitler, or Jesus or Mohammed; those people changed the face of the world because they thought something. And those people were human, constrained by evolution to think in certain ways. An AGI doesn't necessarily have to think inside the human box. There is a lot more of "dangerous thought-space" out there than you seem to be accounting for. Your second paragraph: presumably you think it's literally impossible to invent nanotechnology to make yourself omnipotent, but I'm not willing to place that at less likely than 5%. Something clever enough could probably just manipulate the existing social structures to kill all the humans (something very intelligent that truly wanted nuclear war could probably make it happen). Just a little bit of tweaking to smallpox might be able to destroy humanity anyway. Even just the right new religion might do it (witness the great purges by pretty much every religion ever). None of these suggestions are intended as "this is a plausible way of killing those inconvenient humans"; but if each of my suggestions (nanotech, nuclear war, pandemic, grand new idea like a religion) has a five percent chance of working, that's already a 18.5% chance that exactly one of them would work. Once the humans are dead, of course, it has all the time in the world to achieve all its goals.

namelost 9y ago

https://en.wikipedia.org/wiki/Holocene_extinction https://en.wikipedia.org/wiki/Holocene_extinction

lgas 9y ago

Well, most people assume the development of AGI will in short order lead to the rise of superintelligent AGI and then the reason is precedent -- look at how we treat creatures that are less intelligent than we are. If you're lucky you're a dog or a cat and you're kept as a pet and treated as a member of the family, but you're still a pet. If you're not so lucky you're a cow or a pig and used primarily as a food source. If you're even less lucky you're an insect that gets stepped on for wandering too close to the picnic where we're eating the cow and pig while we watch our dog play.

JabavuAdams 9y ago

The Dodos should have feared us. We don't even hate Dodos, but we killed them all anyway.