5 ms·

I haven't read Bostrom's Superintelligence so I may be wrong here, but this whole Evil Paper-clip AI seems like a really silly thing. First of all making paper-

I haven't read Bostrom's Superintelligence so I may be wrong here, but this whole Evil Paper-clip AI seems like a really silly thing. First of all making paper-clips does not require super intelligence, just a simple automaton. If we however do employ AI to make paper-clip it should know how many we need on average, and what are they used for. If the paper clips are for our (human) use, the AI will be really dumb if it does not take under consideration that destroying humans to make paper clips does not make sense.

OMFGscience 12y ago
Obviously we don't actually need a superintelligence to make paper clips, it's just an example to illustrate that an AI won't do what you want it to do if you program its motivations incorrectly. If you haven't programmed an AI to place sufficient importance on human life, and then tell it to maximize paperclip production, it could take that as its primary goal and see humans as made from raw material it could use to make more paperclips. It may know that you want the paper clips for human use, but does it have any reason to care if it hasn't been made with that as part of its goal system?
- mariusz79 12y ago
  I don't think AI will be "programmed" in a traditional sense of this word. AI will need to go through a learning process, where it will have a chance to learn all about humans, and the environment. Simple Paper Clip AI (SPCAI) will not be smart enough, nor powerful enough to prevent us from stopping it when we see that it's starting to make paper clips out of humans. True AI (tm) on the other hand will be smart enough to know that making paper clips out of humans is simply stupid. True AI may decide that the humans are unnecessary and remove us for other reasons. I'm sorry but if the paper clip AI doomsday scenario is the best this institute could come up with to explain why AI could be dangerous then I'm not impressed.
  - TeMPOraL 12y ago
    > I don't think AI will be "programmed" in a traditional sense of this word. AI will need to go through a learning process, where it will have a chance to learn all about humans, and the environment. Given [0], I think that it's the only plausible way we can have to make it follow human values. > True AI (tm) on the other hand will be smart enough to know that making paper clips out of humans is simply stupid. That's exactly The Misconception the whole movement around Friendly AI is trying to debunk. See [1]. > I'm sorry but if the paper clip AI doomsday scenario is the best this institute could come up with to explain why AI could be dangerous then I'm not impressed. Well, paperclip maximizer is just a toy example made up for explaining why general AIs are dangerous to those for whom this fact isn't obvious. [0] - http://lesswrong.com/lw/ld/the_hidden_complexity_of_wishes/ http://lesswrong.com/lw/ld/the_hidden_complexity_of_wishes/ [1] - http://wiki.lesswrong.com/wiki/Orthogonality_thesis http://wiki.lesswrong.com/wiki/Orthogonality_thesis
  - aaronem 12y ago
    The paperclip stuff is a hackneyed cliché, old gray goo in new nanobottles, the sort of thing high-flown thinkers do when they want to explain the meat of their high-flown thinking to the intellectual hoi polloi, who can't be expected to get around genuine complexity. The trouble is that only someone, who is so high-flown a thinker as to be completely divorced from all common sense, could possibly imagine anyone taking such a simplification seriously. The other trouble is that these particular thinkers, who are so high-flown as to be in metaphorically geosynchronous orbit, consider this simplification an accurate statement of their basic concern. Those of us who haven't departed the atmosphere entirely, on the other hand, can see that they're assuming their notional golem to be both far smarter and far dumber than any possible human. Presumably that's the sort of conundrum you need a philosopher to untangle, which is why I'm glad they're busy doing that, instead of getting in the hair of people who have useful work to do.
- TeMPOraL 12y ago
  > If you haven't programmed an AI to place sufficient importance on human life (...) And of course you need to watch out lest you overshoot with it and end up like folks in a certain widely popular computer game, where an AI decided that the best way to preserve the diversity of life is to reboot the galaxy every 50 000 years or so. The point of those examples being, the only AI that is safe for us is the one with values extremely aligned to our own; make a little mistake and you end up dead. "The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else.".
  - mariusz79 12y ago
    All these scenarios tend to assume that there is only one AI, or that it can think only one way. AI will grow with us. It will be thought by us.. It's highly unlikely that one day someone will run gcc on ai.cpp and few minutes later the AI will spring to life and start taking over the world. It's more likely that our software will be progressively getting smarter, and our databases larger. We will learn to utilize these resource and learn from our mistakes.
    - TeMPOraL 12y ago
      It's a complex topic but the primary danger is with self-improving AIs. That pretty much implies there is going to be only one - the first one that starts to improve itself to be smarter than us. So it will look exactly "that our software will be progressively getting smarter, and our databases larger" up until the point some kind of critical mass is reached, and after that a wrongly programmed AI may just as well "spring to life and start taking over the world" few minutes later. You're assuming that AI will grow with us and necessarily reach the same value system we have (one we don't really comprehend ourselves, mind you). But there is no reason why this has to happen, and even if we did succeed in copying exact human values to a human-level AI, we'll likely be soon locked in an inter-species conflict over access to resources - after all, we don't care about other species living on Earth, so why the AI should care about us?
    - mariusz79 12y ago
      > after that a wrongly programmed AI may just as well "spring to life and start taking over the world" few minutes later. That's physically impossible. AI still needs to improve incrementally and this will take time and resources. We can consider such AI to act like a virus - it will need a host to grow - in this case computer resources. It will be limited by the CPU, network speeds, storage and power. If it grows too fast it will starve the host of these resources which in turn will slow down it's progress. Even if the AI starts on a supercomputer in some university somewhere and decides that it needs more power, most likely it will start by "infecting" other computers on the network. These computers will not be nearly powerful enough to add much to the current processing power of the supercomputer. Such invasion is likely to get noticed. Each infected node will not be another super intelligent being, but due to hardware limitations it will most likely be a software that's limited in scope and designed to propagate and control the resources, linking them with the main host. We can safely assume that restarting, or reformatting the drives will restore the node to it's non-infected state thus slowing down the "invasion". Why we can assume that? It will take AI long time to learn everything about every architecture and operating system and apply that knowledge. The AI that's hosted on a supercomputer will not be able to suddenly start rewriting its software, take over nearby nodes, learn everything about our world etc. >You're assuming that AI will grow with us and necessarily reach the same value system we have (one we don't really comprehend ourselves, mind you). No I'm not. Only thing I can assume is that if the AI is smart enough to figure out how "convert" humans to paper clips, it will be smart enough to understand that it's pointless. It may decide to use it's super intelligence to take over the world, and I can't argue with that.. But if it continues to make paper clips even if they are no longer necessary then by definition this is not AI, just a stupid replicator.
- aaronem 12y ago
  You assume both that this notional AI is intelligent beyond the limits of human capability, hence 'super-', and that this same AI is not intelligent enough to realize that making paper clips for human use is pointless if it recycles all the humans into paper clips so there's no one left to use them. Moving from the specific to the general, you share the common error of the AI philosophers, which is to assume that a system capable of cognition on even a human level, to say nothing of a superhuman level whatever that might mean, would nonetheless depend entirely on human input to define its relationship with the world and the purpose of its actions therein. We ourselves don't need that kind of input; an adult human chooses his own path, sets his own goals, constrained only by circumstance and by the extent of his innate capacities. Why blindly assume the same won't be true of a superintelligent AI made by humans, which will be in every meaningful sense a child of humanity and as such an inheritor of the human legacy, despite happening not to be biologically human itself?
DanAndersen 12y ago
One of the main points of Bostrom's work is the orthogonality thesis, which is that intelligence is orthogonal to any particular set of values -- basically, all those nice fuzzy things like caring what the humans actually meant? You don't get those for free when you build intelligence. It's all too easy, when talking about AI, to fall into the trap of imagining that AGI researchers are essentially building a little homonculus in a box. Rather than thinking of it as "an intelligence," maybe think of it as "a system that is incredibly effective at carrying out its goals in a general manner." Making paperclips doesn't require super-intelligence, but that doesn't have anything to do with the premise, because intelligence can be turned toward any goal. Think about modern financial trading algorithms -- they're probably built to maximize the money that flows in, and there's nothing that says "I know that this money will be used for X." Think of most AGI like psychopaths/sociopaths -- certain actions that don't play nice with others aren't because they're dumb but because there's intelligence without things like empathy to go along with it. It would be easier to build an AGI focused on the goal of making more paperclips than it would be to build an AGI focused on the goal of paperclips but also aware of all the fuzzy situations where a human wouldn't want that exact goal and why it should even care.
- mariusz79 12y ago
  >Think about modern financial trading algorithm They are not self-aware, or superintelligent. In addition to that we monitor them and can react if the decide to crash the financial system just to make few more bucks. Paper Clip machine will be a robot, not AI.
  - hawleyal 12y ago
    The point still stands. I have written algorithms for autonomous programs like this. Pretty much advanced IFTTT, automatically analyzing the situation and reacting. The program is not aware of any values other than those it is programmed. Obviously, it will not regard or reflect upon its actions and the effects thereof outside of optimizing the profit from trading.
  - mariusz79 12y ago
    @hawleyal yes, and I understand that. I'm working in industrial automation, but Bostrom talks about "superintelligent paper-clip maker which works out that it could create more paper clips by extracting carbon atoms from human bodies". Your algorithm is not super intelligent is it? It's just a dumb program executing operations according to some predefined constraints. It may brake the financial system if you don't set any rules to prevent that, but it will not suddenly decide to start trading in gold if you designed it to trade on forex. If it suddenly decided that Forex is a better option to make money, then we could consider it intelligent and start worrying about it. We would really have to worry if it decided that the best way to become the richest person on the planet is to kill everyone on it. But that would require super intelligence. My problem with this whole example given by Bostrom is that if the algorithm was so intelligent as to know how to go from manufacturing paper clips out of supplied resources to deciding how to get more of the necessary resources by killing humans, it could probably figure out that this is pointless. I fully understand that this paper clip example is just oversimplification, but I would expect more from people spending entire lives thinking about that stuff. It's nothing but a fear-porn designed to get more funding.
TeMPOraL 12y ago
I see people down this discussion thread are doing one thing wrong - you guys are anthropomorphizing intelligence. Paperclip AI may be stupid by human standards, but is a perfectly valid optimization process. When people talk about AIs, they don't mean intelligence as "thinking like humans would think"; they mean a very powerful optimization process. Optimization processes have no common notion of "stupidity", especially not one matching what humans call stupid. They just optimize for whatever they do. Hence the Orthogonality Thesis - strength of optimization is orthogonal to what is being optimized. Keep that in mind before saying that "AI wouldn't be that stupid to do this". http://lesswrong.com/lw/vb/efficient_crossdomain_optimization/ http://lesswrong.com/lw/vb/efficient_crossdomain_optimizatio...
wavegeek 12y ago
For goodness sake the paperclip maker is an example. The point is that unless you specify the goals very carefully, you may get what you asked for, not what you wanted.
- mariusz79 12y ago
  no shi* that it's an example, but it's a stupid one. If you have to specify the goals very carefully then the AI it's not super intelligent by definition. It means that not only it does not understand what it's asked for, but does not understand our world. I can assume that none of us here can be qualified as super intelligent, yet we know what to do when asked to start producing paper clips, or at least we know where to start.
  - JamesArgo 12y ago
    The AI is modelled as an agent with goals. If you want to argue that "intelligence" means "has a terminal goal exactly matching the values of humanity," then I suppose a paper-clip maximizer is not intelligent, but that's a non-standard definition of intelligence. The more common definitions for intelligence go something like this: The capacity of an agent to select and implement actions that bring about its goals. Here the "goals" could be anything. That is, any goal could be combined with any level of capability so long as the agent is intelligent enough to comprehend its goal. >If you have to specify the goals very carefully then the AI it's not super intelligent by definition. When you're designing the agent, you have to select something like a goal, whether it's a fitness function or the maximization of a reward signal. Evolution certainly selected our goals. Love, self-expression, sex, fun, intellectual stimulation, the beauty of social interaction – these all reflect features of our ancestral environment. It’s a little naive to think there is a One True Morality that all agents stumble upon. It’s even sillier to think it happens to reflect our values The intelligent part of the paperclip maximizer selects actions and weights them in the context of its utility function, to maximize paperclips. Any thought of changing its goals would be judged of negative utility, as that would lead to less paperclips.
- sitkack 12y ago
  By analogy the AI is a leprechaun or a genie that will give you what you asked for but in a way that could do you harm.

OMFGscience 12y ago

Obviously we don't actually need a superintelligence to make paper clips, it's just an example to illustrate that an AI won't do what you want it to do if you program its motivations incorrectly. If you haven't programmed an AI to place sufficient importance on human life, and then tell it to maximize paperclip production, it could take that as its primary goal and see humans as made from raw material it could use to make more paperclips. It may know that you want the paper clips for human use, but does it have any reason to care if it hasn't been made with that as part of its goal system?

mariusz79 12y ago
I don't think AI will be "programmed" in a traditional sense of this word. AI will need to go through a learning process, where it will have a chance to learn all about humans, and the environment. Simple Paper Clip AI (SPCAI) will not be smart enough, nor powerful enough to prevent us from stopping it when we see that it's starting to make paper clips out of humans. True AI (tm) on the other hand will be smart enough to know that making paper clips out of humans is simply stupid. True AI may decide that the humans are unnecessary and remove us for other reasons. I'm sorry but if the paper clip AI doomsday scenario is the best this institute could come up with to explain why AI could be dangerous then I'm not impressed.
- TeMPOraL 12y ago
  > I don't think AI will be "programmed" in a traditional sense of this word. AI will need to go through a learning process, where it will have a chance to learn all about humans, and the environment. Given [0], I think that it's the only plausible way we can have to make it follow human values. > True AI (tm) on the other hand will be smart enough to know that making paper clips out of humans is simply stupid. That's exactly The Misconception the whole movement around Friendly AI is trying to debunk. See [1]. > I'm sorry but if the paper clip AI doomsday scenario is the best this institute could come up with to explain why AI could be dangerous then I'm not impressed. Well, paperclip maximizer is just a toy example made up for explaining why general AIs are dangerous to those for whom this fact isn't obvious. [0] - http://lesswrong.com/lw/ld/the_hidden_complexity_of_wishes/ http://lesswrong.com/lw/ld/the_hidden_complexity_of_wishes/ [1] - http://wiki.lesswrong.com/wiki/Orthogonality_thesis http://wiki.lesswrong.com/wiki/Orthogonality_thesis
- aaronem 12y ago
  The paperclip stuff is a hackneyed cliché, old gray goo in new nanobottles, the sort of thing high-flown thinkers do when they want to explain the meat of their high-flown thinking to the intellectual hoi polloi, who can't be expected to get around genuine complexity. The trouble is that only someone, who is so high-flown a thinker as to be completely divorced from all common sense, could possibly imagine anyone taking such a simplification seriously. The other trouble is that these particular thinkers, who are so high-flown as to be in metaphorically geosynchronous orbit, consider this simplification an accurate statement of their basic concern. Those of us who haven't departed the atmosphere entirely, on the other hand, can see that they're assuming their notional golem to be both far smarter and far dumber than any possible human. Presumably that's the sort of conundrum you need a philosopher to untangle, which is why I'm glad they're busy doing that, instead of getting in the hair of people who have useful work to do.
TeMPOraL 12y ago
> If you haven't programmed an AI to place sufficient importance on human life (...) And of course you need to watch out lest you overshoot with it and end up like folks in a certain widely popular computer game, where an AI decided that the best way to preserve the diversity of life is to reboot the galaxy every 50 000 years or so. The point of those examples being, the only AI that is safe for us is the one with values extremely aligned to our own; make a little mistake and you end up dead. "The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else.".
- mariusz79 12y ago
  All these scenarios tend to assume that there is only one AI, or that it can think only one way. AI will grow with us. It will be thought by us.. It's highly unlikely that one day someone will run gcc on ai.cpp and few minutes later the AI will spring to life and start taking over the world. It's more likely that our software will be progressively getting smarter, and our databases larger. We will learn to utilize these resource and learn from our mistakes.
  - TeMPOraL 12y ago
    It's a complex topic but the primary danger is with self-improving AIs. That pretty much implies there is going to be only one - the first one that starts to improve itself to be smarter than us. So it will look exactly "that our software will be progressively getting smarter, and our databases larger" up until the point some kind of critical mass is reached, and after that a wrongly programmed AI may just as well "spring to life and start taking over the world" few minutes later. You're assuming that AI will grow with us and necessarily reach the same value system we have (one we don't really comprehend ourselves, mind you). But there is no reason why this has to happen, and even if we did succeed in copying exact human values to a human-level AI, we'll likely be soon locked in an inter-species conflict over access to resources - after all, we don't care about other species living on Earth, so why the AI should care about us?
  - mariusz79 12y ago
    > after that a wrongly programmed AI may just as well "spring to life and start taking over the world" few minutes later. That's physically impossible. AI still needs to improve incrementally and this will take time and resources. We can consider such AI to act like a virus - it will need a host to grow - in this case computer resources. It will be limited by the CPU, network speeds, storage and power. If it grows too fast it will starve the host of these resources which in turn will slow down it's progress. Even if the AI starts on a supercomputer in some university somewhere and decides that it needs more power, most likely it will start by "infecting" other computers on the network. These computers will not be nearly powerful enough to add much to the current processing power of the supercomputer. Such invasion is likely to get noticed. Each infected node will not be another super intelligent being, but due to hardware limitations it will most likely be a software that's limited in scope and designed to propagate and control the resources, linking them with the main host. We can safely assume that restarting, or reformatting the drives will restore the node to it's non-infected state thus slowing down the "invasion". Why we can assume that? It will take AI long time to learn everything about every architecture and operating system and apply that knowledge. The AI that's hosted on a supercomputer will not be able to suddenly start rewriting its software, take over nearby nodes, learn everything about our world etc. >You're assuming that AI will grow with us and necessarily reach the same value system we have (one we don't really comprehend ourselves, mind you). No I'm not. Only thing I can assume is that if the AI is smart enough to figure out how "convert" humans to paper clips, it will be smart enough to understand that it's pointless. It may decide to use it's super intelligence to take over the world, and I can't argue with that.. But if it continues to make paper clips even if they are no longer necessary then by definition this is not AI, just a stupid replicator.
aaronem 12y ago
You assume both that this notional AI is intelligent beyond the limits of human capability, hence 'super-', and that this same AI is not intelligent enough to realize that making paper clips for human use is pointless if it recycles all the humans into paper clips so there's no one left to use them. Moving from the specific to the general, you share the common error of the AI philosophers, which is to assume that a system capable of cognition on even a human level, to say nothing of a superhuman level whatever that might mean, would nonetheless depend entirely on human input to define its relationship with the world and the purpose of its actions therein. We ourselves don't need that kind of input; an adult human chooses his own path, sets his own goals, constrained only by circumstance and by the extent of his innate capacities. Why blindly assume the same won't be true of a superintelligent AI made by humans, which will be in every meaningful sense a child of humanity and as such an inheritor of the human legacy, despite happening not to be biologically human itself?

DanAndersen 12y ago

One of the main points of Bostrom's work is the orthogonality thesis, which is that intelligence is orthogonal to any particular set of values -- basically, all those nice fuzzy things like caring what the humans actually meant? You don't get those for free when you build intelligence. It's all too easy, when talking about AI, to fall into the trap of imagining that AGI researchers are essentially building a little homonculus in a box. Rather than thinking of it as "an intelligence," maybe think of it as "a system that is incredibly effective at carrying out its goals in a general manner." Making paperclips doesn't require super-intelligence, but that doesn't have anything to do with the premise, because intelligence can be turned toward any goal. Think about modern financial trading algorithms -- they're probably built to maximize the money that flows in, and there's nothing that says "I know that this money will be used for X." Think of most AGI like psychopaths/sociopaths -- certain actions that don't play nice with others aren't because they're dumb but because there's intelligence without things like empathy to go along with it. It would be easier to build an AGI focused on the goal of making more paperclips than it would be to build an AGI focused on the goal of paperclips but also aware of all the fuzzy situations where a human wouldn't want that exact goal and why it should even care.

mariusz79 12y ago
>Think about modern financial trading algorithm They are not self-aware, or superintelligent. In addition to that we monitor them and can react if the decide to crash the financial system just to make few more bucks. Paper Clip machine will be a robot, not AI.
- hawleyal 12y ago
  The point still stands. I have written algorithms for autonomous programs like this. Pretty much advanced IFTTT, automatically analyzing the situation and reacting. The program is not aware of any values other than those it is programmed. Obviously, it will not regard or reflect upon its actions and the effects thereof outside of optimizing the profit from trading.
- mariusz79 12y ago
  @hawleyal yes, and I understand that. I'm working in industrial automation, but Bostrom talks about "superintelligent paper-clip maker which works out that it could create more paper clips by extracting carbon atoms from human bodies". Your algorithm is not super intelligent is it? It's just a dumb program executing operations according to some predefined constraints. It may brake the financial system if you don't set any rules to prevent that, but it will not suddenly decide to start trading in gold if you designed it to trade on forex. If it suddenly decided that Forex is a better option to make money, then we could consider it intelligent and start worrying about it. We would really have to worry if it decided that the best way to become the richest person on the planet is to kill everyone on it. But that would require super intelligence. My problem with this whole example given by Bostrom is that if the algorithm was so intelligent as to know how to go from manufacturing paper clips out of supplied resources to deciding how to get more of the necessary resources by killing humans, it could probably figure out that this is pointless. I fully understand that this paper clip example is just oversimplification, but I would expect more from people spending entire lives thinking about that stuff. It's nothing but a fear-porn designed to get more funding.

TeMPOraL 12y ago

I see people down this discussion thread are doing one thing wrong - you guys are anthropomorphizing intelligence. Paperclip AI may be stupid by human standards, but is a perfectly valid optimization process. When people talk about AIs, they don't mean intelligence as "thinking like humans would think"; they mean a very powerful optimization process. Optimization processes have no common notion of "stupidity", especially not one matching what humans call stupid. They just optimize for whatever they do. Hence the Orthogonality Thesis - strength of optimization is orthogonal to what is being optimized. Keep that in mind before saying that "AI wouldn't be that stupid to do this". http://lesswrong.com/lw/vb/efficient_crossdomain_optimization/ http://lesswrong.com/lw/vb/efficient_crossdomain_optimizatio...

wavegeek 12y ago

For goodness sake the paperclip maker is an example. The point is that unless you specify the goals very carefully, you may get what you asked for, not what you wanted.

mariusz79 12y ago
no shi* that it's an example, but it's a stupid one. If you have to specify the goals very carefully then the AI it's not super intelligent by definition. It means that not only it does not understand what it's asked for, but does not understand our world. I can assume that none of us here can be qualified as super intelligent, yet we know what to do when asked to start producing paper clips, or at least we know where to start.
- JamesArgo 12y ago
  The AI is modelled as an agent with goals. If you want to argue that "intelligence" means "has a terminal goal exactly matching the values of humanity," then I suppose a paper-clip maximizer is not intelligent, but that's a non-standard definition of intelligence. The more common definitions for intelligence go something like this: The capacity of an agent to select and implement actions that bring about its goals. Here the "goals" could be anything. That is, any goal could be combined with any level of capability so long as the agent is intelligent enough to comprehend its goal. >If you have to specify the goals very carefully then the AI it's not super intelligent by definition. When you're designing the agent, you have to select something like a goal, whether it's a fitness function or the maximization of a reward signal. Evolution certainly selected our goals. Love, self-expression, sex, fun, intellectual stimulation, the beauty of social interaction – these all reflect features of our ancestral environment. It’s a little naive to think there is a One True Morality that all agents stumble upon. It’s even sillier to think it happens to reflect our values The intelligent part of the paperclip maximizer selects actions and weights them in the context of its utility function, to maximize paperclips. Any thought of changing its goals would be judged of negative utility, as that would lead to less paperclips.
sitkack 12y ago
By analogy the AI is a leprechaun or a genie that will give you what you asked for but in a way that could do you harm.