3 ms·

If the AGI were made from a detailed scan of a human brain converted into a physical simulation, or otherwise proved very similar in terms of embodying human va

If the AGI were made from a detailed scan of a human brain converted into a physical simulation, or otherwise proved very similar in terms of embodying human values, then perhaps. But it doesn't look like things are going this way. Humans make a lot of things that range from clever to mundane to horrifying, and I can't think of a single one of them that I would be happy with replacing humans across the surface of the Earth, even if it were very good at maximizing production figures of a paperclip factory or responding helpfully to queries.

drdaeman 3y ago
That’s a hostile intelligence scenario, right? What I wanted to say is that, I, in particular, really don’t care if it is a human trying to harm me, or a machine. Both of those scenarios are so unfavorable, I really don’t see a point in having any preference. So, I guess, I can’t think of any evil human to replace with even just an evil robot either. But I think I’m less imaginative of a machine becoming sentient and hostile, than a human doing so. The former hadn’t happened - and isn’t exactly trivial, given how big this fear is a part of human culture, and how slow and limited first AGI is inevitably going to be. And then we have an abundance of the latter.
- mitthrowaway2 3y ago
  I think it's more like, even if a total psychopath took over the world and turned it into their personal dystopia, I think there would still be more human-goodness in that world than if a paperclip machine took over. Not merely more (nonzero) humans, but also more things that I and other humans value. Things like art, nature, beauty, play, and maybe hope. There would be some things that the psychopath dictator appreciates and values that I also appreciate and value, even if that person is entirely hostile to my existence. A future of silicon and steel, methodically trampling the earth and grinding the solar system into paperclips, simply doesn't contain any of that. So I prefer the dystopia on those grounds.

drdaeman 3y ago

That’s a hostile intelligence scenario, right? What I wanted to say is that, I, in particular, really don’t care if it is a human trying to harm me, or a machine. Both of those scenarios are so unfavorable, I really don’t see a point in having any preference. So, I guess, I can’t think of any evil human to replace with even just an evil robot either. But I think I’m less imaginative of a machine becoming sentient and hostile, than a human doing so. The former hadn’t happened - and isn’t exactly trivial, given how big this fear is a part of human culture, and how slow and limited first AGI is inevitably going to be. And then we have an abundance of the latter.

mitthrowaway2 3y ago
I think it's more like, even if a total psychopath took over the world and turned it into their personal dystopia, I think there would still be more human-goodness in that world than if a paperclip machine took over. Not merely more (nonzero) humans, but also more things that I and other humans value. Things like art, nature, beauty, play, and maybe hope. There would be some things that the psychopath dictator appreciates and values that I also appreciate and value, even if that person is entirely hostile to my existence. A future of silicon and steel, methodically trampling the earth and grinding the solar system into paperclips, simply doesn't contain any of that. So I prefer the dystopia on those grounds.