4 ms·

The comment about hooking up nukes was hyperbole - I know that's not what's being proposed. And I even agree that there could be real risks in building a self

by shasta 15y ago

The comment about hooking up nukes was hyperbole - I know that's not what's being proposed. And I even agree that there could be real risks in building a self improving machine if coupled with self awareness and physical interaction. But the claim of the article is not that we could build dangerously intelligent machines. It's that every intelligent machine we could build would be inherently dangerous. And to me, that's pretty clearly false. In your description of the threat above, you start with the assumption that the super intelligent AI is self aware and wants out of the box we have it in. How do you justify that assumption? I see it as science fiction. The parable of "the three rules" is: don't build super intelligent self aware machines, tell them to optimize ill understood models of the real world, and then give them the capability to implement their decisions. To extrapolate from there to a general concern about intelligent machines is just fuzzy thinking.

bermanoid 15y ago
I assume that an AI is going to want out of the box not because AI will, by default, have any real desire to get out or anything like that, but because someday, somehow, if we can build an AI at all, someone is going to let it out anyway. If it's not the first group to achieve real AI, it's going to be their local government which seizes the code and hands it over to the military, some foreign government that throws a billion dollars at a research project, or some dude in a basement that stole a copy of the code from his university. Someone's going to try to weaponize it, or they're just going to throw it on the Internet to "see what happens", or they'll try to set it up with a singular focus on playing the stock market, etc. Not everyone with access to the program could be trusted to understand how catastrophic it could be to let it out "in the wild". If someone is going to eventually let an AI out of the box almost no matter what we do, then we have to be damn sure that the first one that gets out is friendly, and further, we have to hope that it has enough of a head start so that by the time some asshole out there deploys (possibly accidentally) a dangerous AI, our friendly one will be able to easily fight it off, or at least mitigate the damage it might do. Maybe I'm wrong, and folks like Eliezer actually think that any self improving AI will automatically try to get "out of the box" (there are definitely arguments that if you aren't very careful to make sure a stable desire to stay in the box is in place then a lot could go wrong and the thing might want to get out), but to me it's rather irrelevan,t because once we hit the threshold of being able to build AI, the cat's out of the bag and someone will eventually do something naughty with it. The first mover advantage here is critical, and we're lucky that for now, most governments are not throwing huge amounts of money at the problem, so maybe before they start there will actually be some ideas on how to safely build an unboxed and unrestrained AI that won't decide to tile the universe with a nice uniform checkerboard pattern because a cosmic ray flipped a bit and corrupted part of its utility calculation function...
- shasta 15y ago
  Two things: 1. It still sounds like you're assuming it's self aware and that letting it out of the box makes sense more than letting Wolfram Alpha out of the box makes sense. The intelligence could just be a powerful logic engine that's in particular capable of analyzing its own design and producing a more powerful version. 2. You seem to be conceding the point I was interested in making, which is that an AI could be controllable if designed carefully. You now seem to be arguing "yes, but it will inevitably fall into the wrong hands." That applies to any powerful technology. On the bright side, you could always try to ask the machine for help in determining the best way to avoid that.
bermanoid 15y ago
More briefly, to respond to just what you said here: "It's that every intelligent machine we could build would be inherently dangerous." My view of the matter is that this is absolutely, 100% true. Any intelligent machine we build will eventually be used by someone that does not understand the serious risks involved in using it in various ways. It will be smart people doing this stuff, but people that haven't really considered the unknown dangers. The real problem is that with self-modifying AI (and trust me, even if it doesn't start out self-modifying, someone will try it eventually...), even seemingly innocuous things could be very dangerous - think of the difficult bullshit we go through to do fundamental physics experiments, and then imagine something 1000x smarter than us, but locked in a box... I can see the Hacker News headlines now: - Facebook loses 100k lines of AI code to disgruntled employee, Wikileaks puts it up for download - Show HN: I Hooked Facebook Brain up to my Roomba and taught it to do the Macarena! - Rate my startup: I set up a Hadoop cluster of Brain nodes working to program themselves in Haskell, and I'm calling it a business! - IRC Bot using Facebook Brain and Node.js passes the Turing Test in less than 20 lines of code [and yes, "Facebook Brain" is my personal vision of hell, too]

by shasta 15y ago

bermanoid 15y ago

I assume that an AI is going to want out of the box not because AI will, by default, have any real desire to get out or anything like that, but because someday, somehow, if we can build an AI at all, someone is going to let it out anyway. If it's not the first group to achieve real AI, it's going to be their local government which seizes the code and hands it over to the military, some foreign government that throws a billion dollars at a research project, or some dude in a basement that stole a copy of the code from his university. Someone's going to try to weaponize it, or they're just going to throw it on the Internet to "see what happens", or they'll try to set it up with a singular focus on playing the stock market, etc. Not everyone with access to the program could be trusted to understand how catastrophic it could be to let it out "in the wild". If someone is going to eventually let an AI out of the box almost no matter what we do, then we have to be damn sure that the first one that gets out is friendly, and further, we have to hope that it has enough of a head start so that by the time some asshole out there deploys (possibly accidentally) a dangerous AI, our friendly one will be able to easily fight it off, or at least mitigate the damage it might do. Maybe I'm wrong, and folks like Eliezer actually think that any self improving AI will automatically try to get "out of the box" (there are definitely arguments that if you aren't very careful to make sure a stable desire to stay in the box is in place then a lot could go wrong and the thing might want to get out), but to me it's rather irrelevan,t because once we hit the threshold of being able to build AI, the cat's out of the bag and someone will eventually do something naughty with it. The first mover advantage here is critical, and we're lucky that for now, most governments are not throwing huge amounts of money at the problem, so maybe before they start there will actually be some ideas on how to safely build an unboxed and unrestrained AI that won't decide to tile the universe with a nice uniform checkerboard pattern because a cosmic ray flipped a bit and corrupted part of its utility calculation function...

shasta 15y ago
Two things: 1. It still sounds like you're assuming it's self aware and that letting it out of the box makes sense more than letting Wolfram Alpha out of the box makes sense. The intelligence could just be a powerful logic engine that's in particular capable of analyzing its own design and producing a more powerful version. 2. You seem to be conceding the point I was interested in making, which is that an AI could be controllable if designed carefully. You now seem to be arguing "yes, but it will inevitably fall into the wrong hands." That applies to any powerful technology. On the bright side, you could always try to ask the machine for help in determining the best way to avoid that.

bermanoid 15y ago

More briefly, to respond to just what you said here: "It's that every intelligent machine we could build would be inherently dangerous." My view of the matter is that this is absolutely, 100% true. Any intelligent machine we build will eventually be used by someone that does not understand the serious risks involved in using it in various ways. It will be smart people doing this stuff, but people that haven't really considered the unknown dangers. The real problem is that with self-modifying AI (and trust me, even if it doesn't start out self-modifying, someone will try it eventually...), even seemingly innocuous things could be very dangerous - think of the difficult bullshit we go through to do fundamental physics experiments, and then imagine something 1000x smarter than us, but locked in a box... I can see the Hacker News headlines now: - Facebook loses 100k lines of AI code to disgruntled employee, Wikileaks puts it up for download - Show HN: I Hooked Facebook Brain up to my Roomba and taught it to do the Macarena! - Rate my startup: I set up a Hadoop cluster of Brain nodes working to program themselves in Haskell, and I'm calling it a business! - IRC Bot using Facebook Brain and Node.js passes the Turing Test in less than 20 lines of code [and yes, "Facebook Brain" is my personal vision of hell, too]