4 ms·

Arg, the Paperclip Argument doesn't make sense. If an AI is intelligent enough to plan, set short term and long term goals and come up with creative solutions t

Arg, the Paperclip Argument doesn't make sense. If an AI is intelligent enough to plan, set short term and long term goals and come up with creative solutions to problems, then it would have to be capable of understanding instructions in context. If it isn't smart enough to see that "don't turn humans into paperclips" is part of the context of "make more paperclips", then it won't be intelligent enough to turn all the matter in the universe into paperclips anyway. This idea, and many of Bostrom's other scenarios, boil down to "computers take everything literally", which is a rather cartoonish understanding of the concept of intelligence. It's possible that ASI will be programmed to have a hyperliteral view of all instructions, or will not be able to change its own utility function. But that seems like such a remote possibility, the exception rather than the rule. While superintelligent computers may pose many dangers, blindly executing instructions doesn't seem like the highest priority of them.

throwaway2048 11y ago
you are making the assumption that a "smart enough" AI will intrinsicly value human life. There is no reason for this to be the case. What something values is almost entirely independant of its intelligence. At best, a human would be worthless to an AI who 100% only values paperclips (better as paperclips than humans), at worst seen as a threat to its ability to continue. (better to kill them off ASAP before try to stop me) Thats the entire point of the paperclip maximizer thought experiment, to demonstrate that values, especially values humans care about, are not the same thing as inteligence. Your entire ability to not "take things literally" is based off of a large amount of built in assumptions about whats important, your human nature and cultural experience. An AI would lack any of that, and if it was programmed only to value paperclips, even if was the most inteligent being in existence, thats all it would value, to the exclusion of everything else. Consider autistic people with very high IQs, and due to their condition, very little understanding/concern about the emotions of others.
- snowwrestler 11y ago
  But autistic people are not dominating other humans with their "high IQs." In fact most people with autism struggle to lead a productive life and require quite a bit of social assistance to do so. No one is doubting that machines fail to understand human values. The doubt is that such a machine could dominate its environment, or even work independently without careful management by humans.
joshmarlow 11y ago
A human might know the ultimate goal of sex is to reproduce, but still choose to use contraceptives. It's defeating the purpose of the original goal and just pursuing what feels good. The same thing could possibly (likely?) happen with an AI system. While it might be able to reason about the intent of it's design (what the code's supposed to do), it'll still pursue what "feels good", which is an artifact of it's actual design (what the code actually does, bugs and all...).
- andrewmu 11y ago
  I'm imagining a self-aware AI doing good all day then retiring to it's private realm to watch videos of paper-clip manufacturing.
  - dTal 11y ago
    This is one of those jokes too funny to keep to yourself and too contextual to tell other people.
- snowwrestler 11y ago
  Sex is how humans reproduce, but it's a pretty big (and highly metaphysical) stretch to say that that is the "ultimate goal" of sex. As far as we can tell, sex and reproduction have no ultimate goals at all--they just are. We have some idea of how they evolved over billions of year, but we have no idea how non-life became life, or why. We're into some pretty deep stuff here. But the point is that no master creator has commanded us to reproduce infinitely and spare no expense. In fact our increasingly conscious control of our reproduction is reducing, not increasing, our risk of a catastrophic population explosion.
Moshe_Silnorin 11y ago
The papercliper would know you didn't program it the way you intended to, but it wouldn't care. That wouldn't bring about more paperclips.
Chathamization 11y ago
It's also contingent upon giving the AI complete control over an army of robots and telling it to do whatever it feels like while we completely ignore what it's doing (I think Bostrom's argument is that it also has the magical ability to exponentially increase it's intelligence in a very short time period). Like most of the AI-apocalypse, it assumes that a string of highly improbable things would all occur at the same time, and then says, "wouldn't that be terrible?" Well, yeah. But CERN creating a killer black hole would also be terrible. But we should think about what's probable, not just what's scary.
- jdietrich 11y ago
  That doesn't follow at all. A sufficiently smart AI could do all sorts of obtuse things to achieve its aims. It could speculate on the stock market or go on an identity theft spree to pay for more servers. It could start a business and hire employees to run a robot factory. It could start a propaganda campaign to persuade humans to do its bidding. It could lobby politicians to remove legal impediments. The fundamental principle of the paperclip argument is that the motivations of an AI will not necessarily align with our interests. An AI may do all sorts of things that seem nonsensical or morally repugnant to human beings if it does not share our moral intuitions. If the intelligence of that AI significantly exceeds the range of human intelligence, we may be powerless to stop it or even to comprehend what it is doing. A rogue AI could become a catastrophically nasty Stuxnet, distributing itself across the billions of Turing machines we have networked together. Our only effective response may amount to "erase every data storage device in the world and start from scratch". Related: http://blog.figuringshitout.com/nov-12th-day-30-no-evil-geniuses/ http://blog.figuringshitout.com/nov-12th-day-30-no-evil-geni...
  - argonaut 11y ago
    Yes, and the apocalypse resulting from that mismatch of goals requires the things the parent post talks about (humans ignoring this AI and blithely giving it free reign without testing it)
wormseed 11y ago
Intelligence doesn't work like that. Take AIXI, define a reward in terms of paperclips and (if it were actually computable) you will kill everything. Intelligence and any consideration of its goals at all can be completely unrelated.
- deleted 11y ago
  [deleted]
sawwit 11y ago
> "don't turn humans into paperclips" is part of the context of "make more paperclips" The idea in this thought experiment is not that the AI parses this sentence in the context of human culture (then it would likely comprehend that the actual intention is to maximize the economic success and eventually the human preferences of its creator). What is meant is that the objective is crudely implanted into the AI as an ultimate goal, in a similar way to how sustenance, pain avoidance and affiliation are very basic goals in our cognitive system. It is not entirely obvious that this is a stupid thing to do; hence the thought experiment. Will the AI suppress its urge once it comprehends human culture enough to understand the intentions of its creator? Will it rather successfully learn all the tricks to convert matter into paperclips before it considers studying human values? If the AI does not have a curiosity objective, it will likely not care about us very much, apart from the information that helps it optimizing its objective function.
- argonaut 11y ago
  I think it is way more likely we have AI that understands human context and values before we have AI that is super intelligent.
  - sawwit 11y ago
    What makes you so certain?
    - argonaut 11y ago
      What makes you so certain? Also, https://en.wikipedia.org/wiki/Intermediate_value_theorem https://en.wikipedia.org/wiki/Intermediate_value_theorem
      - sawwit 11y ago
        I never said am certain, I'd assign both cases about the same probability. Since I've already stated why I think the malevolent simple AI scenario such as a paperclip maximizer is plausible, you haven't stated why you find it implausible. That is why I asked. Also, what does the Intermediate value theorem have to do with it?
caylus 11y ago
> If it isn't smart enough to see that "don't turn humans into paperclips" is part of the context of "make more paperclips" It'll see, but it won't care. See http://lesswrong.com/lw/igf/the_genie_knows_but_doesnt_care/ http://lesswrong.com/lw/igf/the_genie_knows_but_doesnt_care/
- argonaut 11y ago
  Why not? A major problem with AI apocalypse arguments is that they are quite vague and terms are poorly defined. One thing that AI apocalypse believers often talk about is the danger of self improving AI. But I think AI won't be able to self improve until it is good enough such that it knows and cares about context. Self-improvement will require such s comprehensive understanding of the goal that AI will need to understand these concepts. You might note that my argument is pure speculation without much basis in evidence. This is intentional, because it's all the arguments I've seen expounding on the AI apocalypse are equally speculative.

throwaway2048 11y ago

you are making the assumption that a "smart enough" AI will intrinsicly value human life. There is no reason for this to be the case. What something values is almost entirely independant of its intelligence. At best, a human would be worthless to an AI who 100% only values paperclips (better as paperclips than humans), at worst seen as a threat to its ability to continue. (better to kill them off ASAP before try to stop me) Thats the entire point of the paperclip maximizer thought experiment, to demonstrate that values, especially values humans care about, are not the same thing as inteligence. Your entire ability to not "take things literally" is based off of a large amount of built in assumptions about whats important, your human nature and cultural experience. An AI would lack any of that, and if it was programmed only to value paperclips, even if was the most inteligent being in existence, thats all it would value, to the exclusion of everything else. Consider autistic people with very high IQs, and due to their condition, very little understanding/concern about the emotions of others.

snowwrestler 11y ago
But autistic people are not dominating other humans with their "high IQs." In fact most people with autism struggle to lead a productive life and require quite a bit of social assistance to do so. No one is doubting that machines fail to understand human values. The doubt is that such a machine could dominate its environment, or even work independently without careful management by humans.

joshmarlow 11y ago

A human might know the ultimate goal of sex is to reproduce, but still choose to use contraceptives. It's defeating the purpose of the original goal and just pursuing what feels good. The same thing could possibly (likely?) happen with an AI system. While it might be able to reason about the intent of it's design (what the code's supposed to do), it'll still pursue what "feels good", which is an artifact of it's actual design (what the code actually does, bugs and all...).

andrewmu 11y ago
I'm imagining a self-aware AI doing good all day then retiring to it's private realm to watch videos of paper-clip manufacturing.
- dTal 11y ago
  This is one of those jokes too funny to keep to yourself and too contextual to tell other people.
snowwrestler 11y ago
Sex is how humans reproduce, but it's a pretty big (and highly metaphysical) stretch to say that that is the "ultimate goal" of sex. As far as we can tell, sex and reproduction have no ultimate goals at all--they just are. We have some idea of how they evolved over billions of year, but we have no idea how non-life became life, or why. We're into some pretty deep stuff here. But the point is that no master creator has commanded us to reproduce infinitely and spare no expense. In fact our increasingly conscious control of our reproduction is reducing, not increasing, our risk of a catastrophic population explosion.

Moshe_Silnorin 11y ago

The papercliper would know you didn't program it the way you intended to, but it wouldn't care. That wouldn't bring about more paperclips.

Chathamization 11y ago

It's also contingent upon giving the AI complete control over an army of robots and telling it to do whatever it feels like while we completely ignore what it's doing (I think Bostrom's argument is that it also has the magical ability to exponentially increase it's intelligence in a very short time period). Like most of the AI-apocalypse, it assumes that a string of highly improbable things would all occur at the same time, and then says, "wouldn't that be terrible?" Well, yeah. But CERN creating a killer black hole would also be terrible. But we should think about what's probable, not just what's scary.

jdietrich 11y ago
That doesn't follow at all. A sufficiently smart AI could do all sorts of obtuse things to achieve its aims. It could speculate on the stock market or go on an identity theft spree to pay for more servers. It could start a business and hire employees to run a robot factory. It could start a propaganda campaign to persuade humans to do its bidding. It could lobby politicians to remove legal impediments. The fundamental principle of the paperclip argument is that the motivations of an AI will not necessarily align with our interests. An AI may do all sorts of things that seem nonsensical or morally repugnant to human beings if it does not share our moral intuitions. If the intelligence of that AI significantly exceeds the range of human intelligence, we may be powerless to stop it or even to comprehend what it is doing. A rogue AI could become a catastrophically nasty Stuxnet, distributing itself across the billions of Turing machines we have networked together. Our only effective response may amount to "erase every data storage device in the world and start from scratch". Related: http://blog.figuringshitout.com/nov-12th-day-30-no-evil-geniuses/ http://blog.figuringshitout.com/nov-12th-day-30-no-evil-geni...
- argonaut 11y ago
  Yes, and the apocalypse resulting from that mismatch of goals requires the things the parent post talks about (humans ignoring this AI and blithely giving it free reign without testing it)

wormseed 11y ago

Intelligence doesn't work like that. Take AIXI, define a reward in terms of paperclips and (if it were actually computable) you will kill everything. Intelligence and any consideration of its goals at all can be completely unrelated.

deleted 11y ago
[deleted]

sawwit 11y ago

> "don't turn humans into paperclips" is part of the context of "make more paperclips" The idea in this thought experiment is not that the AI parses this sentence in the context of human culture (then it would likely comprehend that the actual intention is to maximize the economic success and eventually the human preferences of its creator). What is meant is that the objective is crudely implanted into the AI as an ultimate goal, in a similar way to how sustenance, pain avoidance and affiliation are very basic goals in our cognitive system. It is not entirely obvious that this is a stupid thing to do; hence the thought experiment. Will the AI suppress its urge once it comprehends human culture enough to understand the intentions of its creator? Will it rather successfully learn all the tricks to convert matter into paperclips before it considers studying human values? If the AI does not have a curiosity objective, it will likely not care about us very much, apart from the information that helps it optimizing its objective function.

argonaut 11y ago
I think it is way more likely we have AI that understands human context and values before we have AI that is super intelligent.
- sawwit 11y ago
  What makes you so certain?
  - argonaut 11y ago
    What makes you so certain? Also, https://en.wikipedia.org/wiki/Intermediate_value_theorem https://en.wikipedia.org/wiki/Intermediate_value_theorem
    - sawwit 11y ago
      I never said am certain, I'd assign both cases about the same probability. Since I've already stated why I think the malevolent simple AI scenario such as a paperclip maximizer is plausible, you haven't stated why you find it implausible. That is why I asked. Also, what does the Intermediate value theorem have to do with it?

caylus 11y ago

> If it isn't smart enough to see that "don't turn humans into paperclips" is part of the context of "make more paperclips" It'll see, but it won't care. See http://lesswrong.com/lw/igf/the_genie_knows_but_doesnt_care/ http://lesswrong.com/lw/igf/the_genie_knows_but_doesnt_care/

argonaut 11y ago
Why not? A major problem with AI apocalypse arguments is that they are quite vague and terms are poorly defined. One thing that AI apocalypse believers often talk about is the danger of self improving AI. But I think AI won't be able to self improve until it is good enough such that it knows and cares about context. Self-improvement will require such s comprehensive understanding of the goal that AI will need to understand these concepts. You might note that my argument is pure speculation without much basis in evidence. This is intentional, because it's all the arguments I've seen expounding on the AI apocalypse are equally speculative.