4 ms·
The superintelligence would likely be smart enough to talk its way around the airgap, see e.g. http://www.yudkowsky.net/singularity/aibox/ http://www.yudkowsky.
by aamar 12y ago
The superintelligence would likely be smart enough to talk its way around the airgap, see e.g. http://www.yudkowsky.net/singularity/aibox/ http://www.yudkowsky.net/singularity/aibox/.
Or, if it had sufficient "motivation"—not necessarily intrinsic motivation, it could be motivation derived from a single poorly-formed request or bad line of code—it could escape by embedding allied code in its output, such as the processor you had it design. The ally could probably be sufficiently buried to evade manual detection.
- gamegoblin 12y agoYes, my idea doesn't take into account potentially malicious AIs. I imagine a thought process of an AI could be something like: 1. I'm to design a processor, OK let's go 2. Hmmm it'd be most efficient for the processors to be able to run an instance of myself to do super-intelligent task scheduling 3. I'll embed some AI on these processors 4. OK humans here's your design! Assuming the AI didn't maliciously try to hide it, a human would probably find that. I suspect most of these "AI exterminates all humans" are a result of simple oversights like this, rather than outright malice. You run into problems, though, when the AI thinks of step 3.5 which is: "If the humans see this change, they won't allow it, which would make for less efficient processors, and I was requested to build the most efficient processor! Therefore I should hide the change as cleverly as possible." So yeah, you're right. Even airgapping has issues.