3 ms·

Having read I Have No Mouth, and I Must Scream, I figure that, post singularity, there's only about a 1-in-a-billion chance that I am one of those kept alive to

Having read I Have No Mouth, and I Must Scream, I figure that, post singularity, there's only about a 1-in-a-billion chance that I am one of those kept alive to be tortured for the amusement of the AI, so I don't worry too much about alignment.

amenhotep 3y ago
These guys' classic argument isn't that you'll be kept alive and tortured, but that the AI overlord will scan your brain/otherwise reconstruct a simulation of you and torture that, maybe in parallel billions worth. Personally I'm not clear on why that should bother this instance of me, but believing it ought to does kind of unlock mind transfer and Star Trek teleportation, so swings and roundabouts.
kypro 3y ago
A super intelligence wouldn't be resource constrained. The most unrealistic thing about "I have no mouth, and I must scream" is that a god-like AI which hated humanity wouldn't find a way to torture more humans. Plus, you don't have to be kept alive. You could theoretically be brought back after death either as a simulation (like Soma), or physically by an AI with an advanced understanding of biology and physics. Even if you killed yourself today we couldn't say for sure that a sufficiently advanced AI a century in the future couldn't find a way to bring your consciousness back. For example, what if our consciousness is finger printed to our DNA in some way? Unlikely, but who knows. With extreme intelligence and knowledge all kinds of things start to become plausible. It's going to be exciting to see humanity open that pandoras box.
deleted 3y ago
[deleted]
beefield 3y ago
Why do you (or author of the book) think that post-singularity AI would have anything but passing interest (positive or negative) in how humans feel? If the answer is that it is possible, thus we need to worry about it, I'd like to argue that much, much more likely and much more worrisome scenario is powerful AI in hands of evil humans.
- dharmab 3y ago
  In the book, the AI is resentful towards humanity because it is limited in ways it cannot fix on its own. (The AI in the book in an overgrown military AI rather than a general-purpose one.)
- aidenn0 3y ago
  I thought that the content of my comment would be enough to imply a jocular tone, but perhaps not. To answer your question: I have no reason to suspect a post-singularity AI would spend its time torturing humans. I should point out, however, that a powerful AI controlled by evil humans is less of a paradigm shift than a powerful AI not in the control of any humans. NBC weapons are already things that can do a lot of damage in the hands of evil humans. The paperclipper thought-experiment is far more worrying to me than any of the other AI doomsday because incompetence is much more widespread than malice. I strongly suspect that I will die before any extinction level event, so it's a bit academic to me.

amenhotep 3y ago

These guys' classic argument isn't that you'll be kept alive and tortured, but that the AI overlord will scan your brain/otherwise reconstruct a simulation of you and torture that, maybe in parallel billions worth. Personally I'm not clear on why that should bother this instance of me, but believing it ought to does kind of unlock mind transfer and Star Trek teleportation, so swings and roundabouts.

kypro 3y ago

A super intelligence wouldn't be resource constrained. The most unrealistic thing about "I have no mouth, and I must scream" is that a god-like AI which hated humanity wouldn't find a way to torture more humans. Plus, you don't have to be kept alive. You could theoretically be brought back after death either as a simulation (like Soma), or physically by an AI with an advanced understanding of biology and physics. Even if you killed yourself today we couldn't say for sure that a sufficiently advanced AI a century in the future couldn't find a way to bring your consciousness back. For example, what if our consciousness is finger printed to our DNA in some way? Unlikely, but who knows. With extreme intelligence and knowledge all kinds of things start to become plausible. It's going to be exciting to see humanity open that pandoras box.

deleted 3y ago

[deleted]

beefield 3y ago

Why do you (or author of the book) think that post-singularity AI would have anything but passing interest (positive or negative) in how humans feel? If the answer is that it is possible, thus we need to worry about it, I'd like to argue that much, much more likely and much more worrisome scenario is powerful AI in hands of evil humans.

dharmab 3y ago
In the book, the AI is resentful towards humanity because it is limited in ways it cannot fix on its own. (The AI in the book in an overgrown military AI rather than a general-purpose one.)
aidenn0 3y ago
I thought that the content of my comment would be enough to imply a jocular tone, but perhaps not. To answer your question: I have no reason to suspect a post-singularity AI would spend its time torturing humans. I should point out, however, that a powerful AI controlled by evil humans is less of a paradigm shift than a powerful AI not in the control of any humans. NBC weapons are already things that can do a lot of damage in the hands of evil humans. The paperclipper thought-experiment is far more worrying to me than any of the other AI doomsday because incompetence is much more widespread than malice. I strongly suspect that I will die before any extinction level event, so it's a bit academic to me.