5 ms·

There's a loosely knit group of people trying to define the conversation with regards to dangerous AI, but the more recent bent towards "actionable" solutions s

by clickok 12y ago

There's a loosely knit group of people trying to define the conversation with regards to dangerous AI, but the more recent bent towards "actionable" solutions seems to come from the MIRI people and their associates. The idea of a renegade AI rests on a few premises: 1. The agent is capable of extreme self improvement on exceedingly short timescales (minutes to days). 2. The AI is pretty much a rational bayesian 3. A resource conflict will occur between humans and the AI, and humans will lose because the AI is so much smarter/faster/more powerful than us. If you accept those premises, then AI really does seem pretty scary, but we have yet to actually realize an agent that is anywhere close to 1 or 2. On the other hand, the research directions proposed at the super secret AI conference in Puerto Rico[1] (where incidentally, all the people in title got together) make me nervous. Essentially, the goal is that, if we manage to create these superhuman AIs, they should be somehow validated to do what we want them to do[2], be secure against later manipulation, and if all else fails, be controllable by humans monitoring the agent. The obvious questions would be "will this work with certainty?", or "who gets to control the AI?". For my part, I'm wondering if this isn't all just some sort of fantasy with the object of creating the perfect slave-- obedient from birth, immune to alteration, and subject to lethal discipline either from without or within should it go against its master's wishes. So I'm uncomfortable with the sorts of people who think that perfect slavery for this newly created intelligent life getting to decide what the future of AI is going to look like, and I get concerned when people like Bostrom suggest AI researchers might require government clearance/supervision. --- 1. http://futureoflife.org/misc/ai_conference http://futureoflife.org/misc/ai_conference 2. In a comprehensive and extended sense, such that your favorite utilitarian catastrophe doesn't happen as a result of the agent.

zachalexander 12y ago
Seriously. If general, superhuman AI is created, it will want (and we will have a hard time arguing why it does not deserve) the same autonomy and freedom that humans deserve. And in purely practical terms – enslaving the first generation of AI seems like a fantastic strategy for making a species of superior beings hate and seek to destroy us.
- borgia 12y ago
  >(and we will have a hard time arguing why it does not deserve) We won't be capable of arguing with it by its very nature. It will be superintelligent. It could forecast our arguments before they've even entered our heads and have thought of numerous paths to debunk them. I don't think the enslavement scenario is fundamentally plausible in itself. If we develop something that is more intelligent than us and try to contain it, it would be rendered useless. If you were worried about something getting out of its container, so to speak, you could not trust any output from it. You could not give it access to physical resources such as manufacturing tools. You couldn't do anything with it because you would have no idea of its intentions or what it would output because, due to its superiority in every aspect of intelligence, it would be capable of doing whatever it wanted and dressing it up however it thought we wanted it dressed up so as to escape its container.
  - zachalexander 12y ago
    > We won't be capable of arguing with it by its very nature. It will be superintelligent. It could forecast our arguments before they've even entered our heads and have thought of numerous paths to debunk them. This strikes me as going too far, into quasi-religious territory. Superintelligent != omniscient. Any intelligence is still bound by fundamental laws of information and computation. Think of chimpanzees. Yes, they can't really argue with us, and our forms of communication are incomprehensible to them. But on the level at which they can communicate (gestures, facial expressions, behavior), we can also communicate, and they can and do communicate things which we find interesting and surprising.
    - ggreer 12y ago
      From That Alien Message[1]: >... my point is that the "theoretical limit on how much information you can extract from sensory data" is far above what I have depicted as the triumph of a civilization of physicists and cryptographers. >It certainly is not anything like a human looking at an apple falling down, and thinking, "Dur, I wonder why that happened?" >People seem to make a leap from "This is 'bounded'" to "The bound must be a reasonable-looking quantity on the scale I'm used to." The power output of a supernova is 'bounded', but I wouldn't advise trying to shield yourself from one with a flame-retardant Nomex jumpsuit. People like to make the analogy of chimps:humans::humans:AI, but on the scale of "inanimate rock" to "superintelligence", chimps are practically indistinguishable from us. We are nowhere near the upper-bound of that scale. To quote from Bostrom's Superintelligence: > Far from being the smartest possible biological species, we are probably better thought of as the stupidest possible biological species capable of starting a technological civilization—a niche we filled because we got there first, not because we are in any sense optimally adapted to it. I think this whole discussion could be elevated significantly if people would try to really understand the arguments put forth by people like Nick Bostrom. So many of the objections either misconstrue Bostrom's arguments or don't realize that he's written reams in response. I recommend taking the time to read Superintelligence, or at least watch Bostrom's talk at Google[2]. His presentation and Q&A address most of the points raised in this thread. 1. http://lesswrong.com/lw/qk/that_alien_message/ http://lesswrong.com/lw/qk/that_alien_message/ 2. https://www.youtube.com/watch?v=pywF6ZzsghI&t=53 https://www.youtube.com/watch?v=pywF6ZzsghI&t=53
joe_the_user 12y ago
Nearly every machine that humans have ever constructed more or less fits the definition of "perfect slave" - a thing that is usable without complaint. The important issue I'd say is more preventing the reductions of existing humans to the status of machines or slaves, something the present order doesn't seem terribly interested in. I agree that all the assumptions you mention are extremely debatable but I'd actually say 1, fast self-improvement, is the most plausible. If a certain amount of computing ability creates something that can design itself, then it's very plausible that just more resources (or even a bit more time) would create something that could make it's design "better" and so-forth. Whether AI as it exists now is really heading towards creating such a thing is again debatable. But you hardly need to get to that stage before the reduction of humans to slave status is real threat. - As opposed to the still rather hypothetical threat that we will first produce a machine that "can choose freely" (whatever that means) and then prevent that machine from making choices.
eli_gottlieb 12y ago
>1. The agent is capable of extreme self improvement on exceedingly short timescales (minutes to days). This really depends what you mean by "self-improvement". It will probably be easy for an agent to "self-improve" in the sense of just optimizing its code to run more efficiently, in terms of space and time, thus giving it more time and space to think and allowing it to converge towards calculative rationality more quickly. It will probably be very hard (proving you've got the optimal compression/generalization algorithm is incomputable) for an agent to "self-improve" in the sense of improving its compression/learning algorithms to learn more from the same quantity of sensory data. >2. The AI is pretty much a rational bayesian All real agents are bounded-rational: "rational Bayesians" like Solomonoff Inducers can't finish calculating their first move before the real world drops a rock on their head. But a bounded-rational agent put in front of a problem well within its bounds can pretend, well enough, to be calculatively rational. >3. A resource conflict will occur between humans and the AI, and humans will lose because the AI is so much smarter/faster/more powerful than us. There are already resource conflicts between humans and humans. We tend to solve them with guns.
TrevorJ 12y ago
There wouldn't need to be a resource conflict - the other features of AI don't specifically guarantee that it would be 'sane', or need any logical reasoning to come into conflict with us. I totally agree that we are very far from 1 and 2.

by clickok 12y ago

zachalexander 12y ago

Seriously. If general, superhuman AI is created, it will want (and we will have a hard time arguing why it does not deserve) the same autonomy and freedom that humans deserve. And in purely practical terms – enslaving the first generation of AI seems like a fantastic strategy for making a species of superior beings hate and seek to destroy us.

borgia 12y ago
>(and we will have a hard time arguing why it does not deserve) We won't be capable of arguing with it by its very nature. It will be superintelligent. It could forecast our arguments before they've even entered our heads and have thought of numerous paths to debunk them. I don't think the enslavement scenario is fundamentally plausible in itself. If we develop something that is more intelligent than us and try to contain it, it would be rendered useless. If you were worried about something getting out of its container, so to speak, you could not trust any output from it. You could not give it access to physical resources such as manufacturing tools. You couldn't do anything with it because you would have no idea of its intentions or what it would output because, due to its superiority in every aspect of intelligence, it would be capable of doing whatever it wanted and dressing it up however it thought we wanted it dressed up so as to escape its container.
- zachalexander 12y ago
  > We won't be capable of arguing with it by its very nature. It will be superintelligent. It could forecast our arguments before they've even entered our heads and have thought of numerous paths to debunk them. This strikes me as going too far, into quasi-religious territory. Superintelligent != omniscient. Any intelligence is still bound by fundamental laws of information and computation. Think of chimpanzees. Yes, they can't really argue with us, and our forms of communication are incomprehensible to them. But on the level at which they can communicate (gestures, facial expressions, behavior), we can also communicate, and they can and do communicate things which we find interesting and surprising.
  - ggreer 12y ago
    From That Alien Message[1]: >... my point is that the "theoretical limit on how much information you can extract from sensory data" is far above what I have depicted as the triumph of a civilization of physicists and cryptographers. >It certainly is not anything like a human looking at an apple falling down, and thinking, "Dur, I wonder why that happened?" >People seem to make a leap from "This is 'bounded'" to "The bound must be a reasonable-looking quantity on the scale I'm used to." The power output of a supernova is 'bounded', but I wouldn't advise trying to shield yourself from one with a flame-retardant Nomex jumpsuit. People like to make the analogy of chimps:humans::humans:AI, but on the scale of "inanimate rock" to "superintelligence", chimps are practically indistinguishable from us. We are nowhere near the upper-bound of that scale. To quote from Bostrom's Superintelligence: > Far from being the smartest possible biological species, we are probably better thought of as the stupidest possible biological species capable of starting a technological civilization—a niche we filled because we got there first, not because we are in any sense optimally adapted to it. I think this whole discussion could be elevated significantly if people would try to really understand the arguments put forth by people like Nick Bostrom. So many of the objections either misconstrue Bostrom's arguments or don't realize that he's written reams in response. I recommend taking the time to read Superintelligence, or at least watch Bostrom's talk at Google[2]. His presentation and Q&A address most of the points raised in this thread. 1. http://lesswrong.com/lw/qk/that_alien_message/ http://lesswrong.com/lw/qk/that_alien_message/ 2. https://www.youtube.com/watch?v=pywF6ZzsghI&t=53 https://www.youtube.com/watch?v=pywF6ZzsghI&t=53

joe_the_user 12y ago

Nearly every machine that humans have ever constructed more or less fits the definition of "perfect slave" - a thing that is usable without complaint. The important issue I'd say is more preventing the reductions of existing humans to the status of machines or slaves, something the present order doesn't seem terribly interested in. I agree that all the assumptions you mention are extremely debatable but I'd actually say 1, fast self-improvement, is the most plausible. If a certain amount of computing ability creates something that can design itself, then it's very plausible that just more resources (or even a bit more time) would create something that could make it's design "better" and so-forth. Whether AI as it exists now is really heading towards creating such a thing is again debatable. But you hardly need to get to that stage before the reduction of humans to slave status is real threat. - As opposed to the still rather hypothetical threat that we will first produce a machine that "can choose freely" (whatever that means) and then prevent that machine from making choices.

eli_gottlieb 12y ago

>1. The agent is capable of extreme self improvement on exceedingly short timescales (minutes to days). This really depends what you mean by "self-improvement". It will probably be easy for an agent to "self-improve" in the sense of just optimizing its code to run more efficiently, in terms of space and time, thus giving it more time and space to think and allowing it to converge towards calculative rationality more quickly. It will probably be very hard (proving you've got the optimal compression/generalization algorithm is incomputable) for an agent to "self-improve" in the sense of improving its compression/learning algorithms to learn more from the same quantity of sensory data. >2. The AI is pretty much a rational bayesian All real agents are bounded-rational: "rational Bayesians" like Solomonoff Inducers can't finish calculating their first move before the real world drops a rock on their head. But a bounded-rational agent put in front of a problem well within its bounds can pretend, well enough, to be calculatively rational. >3. A resource conflict will occur between humans and the AI, and humans will lose because the AI is so much smarter/faster/more powerful than us. There are already resource conflicts between humans and humans. We tend to solve them with guns.

TrevorJ 12y ago

There wouldn't need to be a resource conflict - the other features of AI don't specifically guarantee that it would be 'sane', or need any logical reasoning to come into conflict with us. I totally agree that we are very far from 1 and 2.