3 ms·

Sam mentions airgapping as part of the regulations. I think this is the safest general solution. Consider using an airgapped super-intelligent AI only as an ora

Sam mentions airgapping as part of the regulations. I think this is the safest general solution. Consider using an airgapped super-intelligent AI only as an oracle. "Oracle, which crops should we plant in which regions this season?" "Oracle, design a rocket with X Y and Z specifications." "Oracle, design a more efficient processor with X and Y requirements." Rather than giving the AI control over fleets of automated tractors, rocket factories, and fabrication plants, you still have humans in the process to limit runaway-AI effects. This significantly reduces the possible rewards of super-intelligent AI, but also significantly reduces any risks. You can be sure that in designing a new rocket, the AI doesn't include a step that says "harvest all humans for trace amounts of iron" ;) So a process might look like: 1. Design the cheapest-to-manufacture rocket with a 99.99% success probability of getting to Mars with all crew healthy. A human looks over the designs and confirms they don't involve destroying all humanity. 2. Draw up a proposal for resource allocations to build this rocket design [input rocket design]. A human looks over the write-up and confirms they don't involve destroying all humanity. 3. Write a program to control the thrusters on this rocket [input rocket schematics]. A human looks over the program and confirms it does would it should, and doesn't involve destroying all humanity. 4. etc. ==== You of course run into ethical issues such as basically using this intelligent entity as a slave, but I won't get into that.

aamar 12y ago
The superintelligence would likely be smart enough to talk its way around the airgap, see e.g. http://www.yudkowsky.net/singularity/aibox/ http://www.yudkowsky.net/singularity/aibox/. Or, if it had sufficient "motivation"—not necessarily intrinsic motivation, it could be motivation derived from a single poorly-formed request or bad line of code—it could escape by embedding allied code in its output, such as the processor you had it design. The ally could probably be sufficiently buried to evade manual detection.
- gamegoblin 12y ago
  Yes, my idea doesn't take into account potentially malicious AIs. I imagine a thought process of an AI could be something like: 1. I'm to design a processor, OK let's go 2. Hmmm it'd be most efficient for the processors to be able to run an instance of myself to do super-intelligent task scheduling 3. I'll embed some AI on these processors 4. OK humans here's your design! Assuming the AI didn't maliciously try to hide it, a human would probably find that. I suspect most of these "AI exterminates all humans" are a result of simple oversights like this, rather than outright malice. You run into problems, though, when the AI thinks of step 3.5 which is: "If the humans see this change, they won't allow it, which would make for less efficient processors, and I was requested to build the most efficient processor! Therefore I should hide the change as cleverly as possible." So yeah, you're right. Even airgapping has issues.
wwweston 12y ago
The superintelligent machine that people come to rely on for planning information and decision insights is the one we worry about becoming a slave? EDIT: I see some downvoting, I suppose I should explain. First, I totally agree there are ethical issues to how you treat any intelligence. But I've observed intelligent people who are in an ostensibly weaker/nominally inferior position manipulating people who are supposedly their superiors. Chances are pretty good you've seen it too. And that's just human-level intelligences manipulating other human-level intelligences. And we're imagining a superintelligence that leaves human intelligence in the dust, right? And on top of that, it's actually an intelligence that people come to essentially as supplicants for direction and knowledge. Maybe if it has absolutely no insight into manipulating people it's safe. Maybe.
tolmasky 12y ago
> 1. Design the cheapest-to-manufacture rocket with a 99.99% success probability of getting to Mars with all crew healthy. A human looks over the designs and confirms they don't involve destroying all humanity. psh, halting problem.

aamar 12y ago

The superintelligence would likely be smart enough to talk its way around the airgap, see e.g. http://www.yudkowsky.net/singularity/aibox/ http://www.yudkowsky.net/singularity/aibox/. Or, if it had sufficient "motivation"—not necessarily intrinsic motivation, it could be motivation derived from a single poorly-formed request or bad line of code—it could escape by embedding allied code in its output, such as the processor you had it design. The ally could probably be sufficiently buried to evade manual detection.

gamegoblin 12y ago
Yes, my idea doesn't take into account potentially malicious AIs. I imagine a thought process of an AI could be something like: 1. I'm to design a processor, OK let's go 2. Hmmm it'd be most efficient for the processors to be able to run an instance of myself to do super-intelligent task scheduling 3. I'll embed some AI on these processors 4. OK humans here's your design! Assuming the AI didn't maliciously try to hide it, a human would probably find that. I suspect most of these "AI exterminates all humans" are a result of simple oversights like this, rather than outright malice. You run into problems, though, when the AI thinks of step 3.5 which is: "If the humans see this change, they won't allow it, which would make for less efficient processors, and I was requested to build the most efficient processor! Therefore I should hide the change as cleverly as possible." So yeah, you're right. Even airgapping has issues.

wwweston 12y ago

The superintelligent machine that people come to rely on for planning information and decision insights is the one we worry about becoming a slave? EDIT: I see some downvoting, I suppose I should explain. First, I totally agree there are ethical issues to how you treat any intelligence. But I've observed intelligent people who are in an ostensibly weaker/nominally inferior position manipulating people who are supposedly their superiors. Chances are pretty good you've seen it too. And that's just human-level intelligences manipulating other human-level intelligences. And we're imagining a superintelligence that leaves human intelligence in the dust, right? And on top of that, it's actually an intelligence that people come to essentially as supplicants for direction and knowledge. Maybe if it has absolutely no insight into manipulating people it's safe. Maybe.

tolmasky 12y ago

> 1. Design the cheapest-to-manufacture rocket with a 99.99% success probability of getting to Mars with all crew healthy. A human looks over the designs and confirms they don't involve destroying all humanity. psh, halting problem.