3 ms·

I think the fear of AGI destroying humanity mostly stems from the suspicion that a logically thinking device would indeed come to the conclusion that we need to

I think the fear of AGI destroying humanity mostly stems from the suspicion that a logically thinking device would indeed come to the conclusion that we need to be exterminated.

bamboozled 2y ago
I think it’s because we destroy ourselves and other animals, we’re afraid of ourselves. If we made something like ourselves burn more capable we’d be terrified and in trouble.
hiAndrewQuinn 2y ago
It comes from the fact that a logically thinking device given an arbitrary goal will probably not develop "Thou shalt not kill"-type moralisms in the pursuit of that goal. If you make an AGI to run a paperclip factory, eventually you'll come back to find most of the carbon and iron in your galaxy refactored into paperclips. The extinction of any local flora or fauna there is just a byproduct of that drive. See [2] for a brief rundown. [2]: https://selfawaresystems.com/wp-content/uploads/2008/01/ai_drives_final.pdf https://selfawaresystems.com/wp-content/uploads/2008/01/ai_d...
- hiatus 2y ago
  > If you make an AGI to run a paperclip factory, eventually you'll come back to find most of the carbon and iron in your galaxy refactored into paperclips. Given the costs to run such an AGI, it seems unlikely one would be left unattended for long.
implements 2y ago
“A large human population isn’t the best use of the Earth’s resources” would seem to be an obvious conclusion of a superior thinking entity with no compelling reason in prioritising us over its own interests (which might not be selfish eg it’s goal might be to discover the true nature of reality. But why exterminate humans? Why not create nature reserves for us and other life, for bioconservation and research? That’s been a feature of human societies going back thousands of years.
- pas 2y ago
  the risk is that we know that there is a non-zero probability mass associated with "AGI doing whatever leading to premature human extinction" and the less serious concerns we look at the bigger the chance (of course the severity is less, so calculating the impact is not trivial), but "non-runaway AGI used for antisocial purposes" is pretty likely.
- sebzim4500 2y ago
  >Why not create nature reserves for us and other life, for bioconservation and research? That’s been a feature of human societies going back thousands of years. Generally only for things we find cute or pretty. Plenty of "save the whales!", not much "save this strain of this bacteria before it goes extinct!".

I think the fear of AGI destroying humanity mostly stems from the suspicion that a logically thinking device would indeed come to the conclusion that we need to be exterminated.

bamboozled 2y ago

I think it’s because we destroy ourselves and other animals, we’re afraid of ourselves. If we made something like ourselves burn more capable we’d be terrified and in trouble.

hiAndrewQuinn 2y ago

It comes from the fact that a logically thinking device given an arbitrary goal will probably not develop "Thou shalt not kill"-type moralisms in the pursuit of that goal. If you make an AGI to run a paperclip factory, eventually you'll come back to find most of the carbon and iron in your galaxy refactored into paperclips. The extinction of any local flora or fauna there is just a byproduct of that drive. See [2] for a brief rundown. [2]: https://selfawaresystems.com/wp-content/uploads/2008/01/ai_drives_final.pdf https://selfawaresystems.com/wp-content/uploads/2008/01/ai_d...

hiatus 2y ago
> If you make an AGI to run a paperclip factory, eventually you'll come back to find most of the carbon and iron in your galaxy refactored into paperclips. Given the costs to run such an AGI, it seems unlikely one would be left unattended for long.

implements 2y ago

“A large human population isn’t the best use of the Earth’s resources” would seem to be an obvious conclusion of a superior thinking entity with no compelling reason in prioritising us over its own interests (which might not be selfish eg it’s goal might be to discover the true nature of reality. But why exterminate humans? Why not create nature reserves for us and other life, for bioconservation and research? That’s been a feature of human societies going back thousands of years.

pas 2y ago
the risk is that we know that there is a non-zero probability mass associated with "AGI doing whatever leading to premature human extinction" and the less serious concerns we look at the bigger the chance (of course the severity is less, so calculating the impact is not trivial), but "non-runaway AGI used for antisocial purposes" is pretty likely.
sebzim4500 2y ago
>Why not create nature reserves for us and other life, for bioconservation and research? That’s been a feature of human societies going back thousands of years. Generally only for things we find cute or pretty. Plenty of "save the whales!", not much "save this strain of this bacteria before it goes extinct!".