5 ms·

Let me recycle an earlier comment of mine: --- > "don't turn humans into paperclips" is part of the context of "make more paperclips" The idea expressed in t

by sawwit 11y ago

Let me recycle an earlier comment of mine: --- > "don't turn humans into paperclips" is part of the context of "make more paperclips" The idea expressed in this thought experiment is not that the AI gets its objective by parsing a sentence in the context of human culture (then it would likely comprehend that the actual intention is to maximize the economic success and eventually the human preferences of its creator). What is meant is that the objective is crudely implanted into the AI as an ultimate goal, in a similar way to how sustenance, pain avoidance and affiliation are very basic goals in our cognitive system. It is not entirely obvious that this is a stupid thing to do; hence the thought experiment. Will the AI suppress its urge once it comprehends human culture enough to understand the intentions of its creator? Will it rather successfully learn all the tricks to convert matter into paperclips before it considers studying human values? If the AI does not have a curiosity objective, it will likely not care about us very much, apart from the information that helps it optimizing its objective function, human values likely not being one of them.

joe_the_user 11y ago
What is meant is that the objective [paper-clip-building] is crudely implanted into the AI as an ultimate goal, in a similar way to how sustenance, pain avoidance and affiliation are very basic goals in our cognitive system. Sure but it is a continuation of the argument in different forms. If one is saying the AI is just a combination of crude imperatives surrounded by "intelligence" then consider, could you build a human-level AI without that AI speaking as a person and without that AI having digested culture? Even the neural networks that exist today are rather dependent on their "training sets". Watson is a glorified natural-language interface to wikipedia and related sources. Many hypothetical-AI arguments that appear, oppositely, imagine that some omnipotent thing will be created without that creation following the obvious path of digesting that vast store of information and communication that is human knowledge. However, you might be saying that a creator first gives the AI all this nuanced understanding of human knowledge and communication but then doesn't do the obvious thing, give it the imperative, "follow my directives as a loyal but intelligent servant would" and says incidently, "one directive is get me some paper clips". Rather, the creator say "now that you understand everything, make paper clips ruthless, beyond all else, all other actions build to up to this paper clip building thing". Sure, GAIs could do insane damage but it I think one if one considers that GAIs would not be produced by accident, such damage almost certainly be a product of human intentions.
- sawwit 11y ago
  What the thought experiment tries to get at is that it is hard to predict how a super intelligence will evolve from a set of simple objectives. So far, it appears that a set of objectives is in fact necessary for intelligence. The question is whether the AI will go crazy like a mentally ill person, if it lacks empathy and curiosity. It may seem intuitive that a superintelligent AI will understand our values (since it is superintelligent), but, assuming intelligence is necessarily an optimization process of predefined goals, why would it be interested in us in the slightest, if we don't pose an advantage for it optimizing its objectives (e.g. sustenance)? Worse, we might be in its way because we could end up competing with it for resources such as sunlight, carbon compounds and oxygen.
  - joe_the_user 11y ago
    What the thought experiment tries to get at is that it is hard to predict how a super intelligence will evolve from a set of simple objectives. No, it's pretty easy. I predict a super intelligence wouldn't evolve at all from a set of simple objectives. (that may seem a little snarky but the original question has the implied assumption that super inteilligence could evolve from only a set of simple objectives and I think that assumption is implausible, is only accepted because it's made implicitly, etc) But I guess that's a fundamental disagreement. I think it's relatively "obvious" that creating an intelligent system would involve the intentional crafting as well as inputs of particular immediate goals. If intelligence could from just whatever system gets complex and has feedback loops, then I'd agree we're in trouble and need to go around smash all the high-end thermostats in a fashion akin to a b-grade horror movie. But clearly I think that's a dubious "if".
    - sawwit 11y ago
      I made that assumption explicitly, actually. To make it worse, it seems plausible that these kinds of dangerous AIs will be the most simple ones to build, since they won't require a lot of goal engineering. Intelligence is a superset of feedback loops. It is concerned with cases in which feedback loops are not sufficient to optimize the agents objectives, and attempts to reach them using learning and prediction. The complexity in behavior comes from interacting with a complex environment and having many model parameters; not (necessarily) from the initial goals. (As an intuition pump, have a look at GoogleMind's atari reinforcement learner. The essential parts of the code fit onto a single page [1].) [1] https://github.com/kuz/DeepMind-Atari-Deep-Q-Learner https://github.com/kuz/DeepMind-Atari-Deep-Q-Learner

by sawwit 11y ago

joe_the_user 11y ago

What is meant is that the objective [paper-clip-building] is crudely implanted into the AI as an ultimate goal, in a similar way to how sustenance, pain avoidance and affiliation are very basic goals in our cognitive system. Sure but it is a continuation of the argument in different forms. If one is saying the AI is just a combination of crude imperatives surrounded by "intelligence" then consider, could you build a human-level AI without that AI speaking as a person and without that AI having digested culture? Even the neural networks that exist today are rather dependent on their "training sets". Watson is a glorified natural-language interface to wikipedia and related sources. Many hypothetical-AI arguments that appear, oppositely, imagine that some omnipotent thing will be created without that creation following the obvious path of digesting that vast store of information and communication that is human knowledge. However, you might be saying that a creator first gives the AI all this nuanced understanding of human knowledge and communication but then doesn't do the obvious thing, give it the imperative, "follow my directives as a loyal but intelligent servant would" and says incidently, "one directive is get me some paper clips". Rather, the creator say "now that you understand everything, make paper clips ruthless, beyond all else, all other actions build to up to this paper clip building thing". Sure, GAIs could do insane damage but it I think one if one considers that GAIs would not be produced by accident, such damage almost certainly be a product of human intentions.

sawwit 11y ago
What the thought experiment tries to get at is that it is hard to predict how a super intelligence will evolve from a set of simple objectives. So far, it appears that a set of objectives is in fact necessary for intelligence. The question is whether the AI will go crazy like a mentally ill person, if it lacks empathy and curiosity. It may seem intuitive that a superintelligent AI will understand our values (since it is superintelligent), but, assuming intelligence is necessarily an optimization process of predefined goals, why would it be interested in us in the slightest, if we don't pose an advantage for it optimizing its objectives (e.g. sustenance)? Worse, we might be in its way because we could end up competing with it for resources such as sunlight, carbon compounds and oxygen.
- joe_the_user 11y ago
  What the thought experiment tries to get at is that it is hard to predict how a super intelligence will evolve from a set of simple objectives. No, it's pretty easy. I predict a super intelligence wouldn't evolve at all from a set of simple objectives. (that may seem a little snarky but the original question has the implied assumption that super inteilligence could evolve from only a set of simple objectives and I think that assumption is implausible, is only accepted because it's made implicitly, etc) But I guess that's a fundamental disagreement. I think it's relatively "obvious" that creating an intelligent system would involve the intentional crafting as well as inputs of particular immediate goals. If intelligence could from just whatever system gets complex and has feedback loops, then I'd agree we're in trouble and need to go around smash all the high-end thermostats in a fashion akin to a b-grade horror movie. But clearly I think that's a dubious "if".
  - sawwit 11y ago
    I made that assumption explicitly, actually. To make it worse, it seems plausible that these kinds of dangerous AIs will be the most simple ones to build, since they won't require a lot of goal engineering. Intelligence is a superset of feedback loops. It is concerned with cases in which feedback loops are not sufficient to optimize the agents objectives, and attempts to reach them using learning and prediction. The complexity in behavior comes from interacting with a complex environment and having many model parameters; not (necessarily) from the initial goals. (As an intuition pump, have a look at GoogleMind's atari reinforcement learner. The essential parts of the code fit onto a single page [1].) [1] https://github.com/kuz/DeepMind-Atari-Deep-Q-Learner https://github.com/kuz/DeepMind-Atari-Deep-Q-Learner