5 ms·

Wow, there's some serious lack of critical thinking going on in this thread That you think people are talking about hooking up the nuke switch to a neural netw

Wow, there's some serious lack of critical thinking going on in this thread That you think people are talking about hooking up the nuke switch to a neural network merely shows your lack of knowledge about how much this topic has been studied. The real concern with the singularity has nothing to do with people controlling it, it has to do with building a self-modifying system that will explore regions of design space that we are too stupid to know anything about. It has to do with that self-modifying system reaching out further into design space that it is too stupid to know anything about. It has to do with the fact that an intelligence that had gotten smarter than us could extremely easily get around any restrictive measures we put in its way, no matter how severe they were (social engineering is how the most effective hackers (in the Bad Guys With Computers sense of the word) always work, there's no reason to think that an AI couldn't social engineer its way out of any box we put around it). An AI getting its hands on a nuke is just about the least terrifying thing that could happen; at least a nuke only blows up if you shoot it at someone. We have no idea the kinds of things that something so much smarter than us might develop or stumble into, and worse, we have no idea if when it is smart enough to develop these things it might still be too stupid to know how to control them. A nuclear bomb is a runaway chain reaction with a hard limit on destructive power; we don't know if there might be other chain reactions that are harder to predict and limit, reactions that a super-intelligent AI might fuck up its damage estimates on. Hell, a simple re-design of an AI's code could quite easily cause a runaway problem if it flips a wrong bit somewhere and changes its goal systems in a bad way (there are lots of ways to come up with seemingly innocuous goal systems that, if optimally achieved by a super-intelligent being, would pretty much wipe out everything that we care about and consider interesting).

shasta 15y ago
The comment about hooking up nukes was hyperbole - I know that's not what's being proposed. And I even agree that there could be real risks in building a self improving machine if coupled with self awareness and physical interaction. But the claim of the article is not that we could build dangerously intelligent machines. It's that every intelligent machine we could build would be inherently dangerous. And to me, that's pretty clearly false. In your description of the threat above, you start with the assumption that the super intelligent AI is self aware and wants out of the box we have it in. How do you justify that assumption? I see it as science fiction. The parable of "the three rules" is: don't build super intelligent self aware machines, tell them to optimize ill understood models of the real world, and then give them the capability to implement their decisions. To extrapolate from there to a general concern about intelligent machines is just fuzzy thinking.
- bermanoid 15y ago
  I assume that an AI is going to want out of the box not because AI will, by default, have any real desire to get out or anything like that, but because someday, somehow, if we can build an AI at all, someone is going to let it out anyway. If it's not the first group to achieve real AI, it's going to be their local government which seizes the code and hands it over to the military, some foreign government that throws a billion dollars at a research project, or some dude in a basement that stole a copy of the code from his university. Someone's going to try to weaponize it, or they're just going to throw it on the Internet to "see what happens", or they'll try to set it up with a singular focus on playing the stock market, etc. Not everyone with access to the program could be trusted to understand how catastrophic it could be to let it out "in the wild". If someone is going to eventually let an AI out of the box almost no matter what we do, then we have to be damn sure that the first one that gets out is friendly, and further, we have to hope that it has enough of a head start so that by the time some asshole out there deploys (possibly accidentally) a dangerous AI, our friendly one will be able to easily fight it off, or at least mitigate the damage it might do. Maybe I'm wrong, and folks like Eliezer actually think that any self improving AI will automatically try to get "out of the box" (there are definitely arguments that if you aren't very careful to make sure a stable desire to stay in the box is in place then a lot could go wrong and the thing might want to get out), but to me it's rather irrelevan,t because once we hit the threshold of being able to build AI, the cat's out of the bag and someone will eventually do something naughty with it. The first mover advantage here is critical, and we're lucky that for now, most governments are not throwing huge amounts of money at the problem, so maybe before they start there will actually be some ideas on how to safely build an unboxed and unrestrained AI that won't decide to tile the universe with a nice uniform checkerboard pattern because a cosmic ray flipped a bit and corrupted part of its utility calculation function...
  - shasta 15y ago
    Two things: 1. It still sounds like you're assuming it's self aware and that letting it out of the box makes sense more than letting Wolfram Alpha out of the box makes sense. The intelligence could just be a powerful logic engine that's in particular capable of analyzing its own design and producing a more powerful version. 2. You seem to be conceding the point I was interested in making, which is that an AI could be controllable if designed carefully. You now seem to be arguing "yes, but it will inevitably fall into the wrong hands." That applies to any powerful technology. On the bright side, you could always try to ask the machine for help in determining the best way to avoid that.

shasta 15y ago

The comment about hooking up nukes was hyperbole - I know that's not what's being proposed. And I even agree that there could be real risks in building a self improving machine if coupled with self awareness and physical interaction. But the claim of the article is not that we could build dangerously intelligent machines. It's that every intelligent machine we could build would be inherently dangerous. And to me, that's pretty clearly false. In your description of the threat above, you start with the assumption that the super intelligent AI is self aware and wants out of the box we have it in. How do you justify that assumption? I see it as science fiction. The parable of "the three rules" is: don't build super intelligent self aware machines, tell them to optimize ill understood models of the real world, and then give them the capability to implement their decisions. To extrapolate from there to a general concern about intelligent machines is just fuzzy thinking.

bermanoid 15y ago
I assume that an AI is going to want out of the box not because AI will, by default, have any real desire to get out or anything like that, but because someday, somehow, if we can build an AI at all, someone is going to let it out anyway. If it's not the first group to achieve real AI, it's going to be their local government which seizes the code and hands it over to the military, some foreign government that throws a billion dollars at a research project, or some dude in a basement that stole a copy of the code from his university. Someone's going to try to weaponize it, or they're just going to throw it on the Internet to "see what happens", or they'll try to set it up with a singular focus on playing the stock market, etc. Not everyone with access to the program could be trusted to understand how catastrophic it could be to let it out "in the wild". If someone is going to eventually let an AI out of the box almost no matter what we do, then we have to be damn sure that the first one that gets out is friendly, and further, we have to hope that it has enough of a head start so that by the time some asshole out there deploys (possibly accidentally) a dangerous AI, our friendly one will be able to easily fight it off, or at least mitigate the damage it might do. Maybe I'm wrong, and folks like Eliezer actually think that any self improving AI will automatically try to get "out of the box" (there are definitely arguments that if you aren't very careful to make sure a stable desire to stay in the box is in place then a lot could go wrong and the thing might want to get out), but to me it's rather irrelevan,t because once we hit the threshold of being able to build AI, the cat's out of the bag and someone will eventually do something naughty with it. The first mover advantage here is critical, and we're lucky that for now, most governments are not throwing huge amounts of money at the problem, so maybe before they start there will actually be some ideas on how to safely build an unboxed and unrestrained AI that won't decide to tile the universe with a nice uniform checkerboard pattern because a cosmic ray flipped a bit and corrupted part of its utility calculation function...
- shasta 15y ago
  Two things: 1. It still sounds like you're assuming it's self aware and that letting it out of the box makes sense more than letting Wolfram Alpha out of the box makes sense. The intelligence could just be a powerful logic engine that's in particular capable of analyzing its own design and producing a more powerful version. 2. You seem to be conceding the point I was interested in making, which is that an AI could be controllable if designed carefully. You now seem to be arguing "yes, but it will inevitably fall into the wrong hands." That applies to any powerful technology. On the bright side, you could always try to ask the machine for help in determining the best way to avoid that.