4 ms·

Check for instance the 'Paperclip maximizer' thought experiment, https://wiki.lesswrong.com/wiki/Paperclip_maximizer https://wiki.lesswrong.com/wiki/Paperclip_m

by jononor 10y ago

Check for instance the 'Paperclip maximizer' thought experiment, https://wiki.lesswrong.com/wiki/Paperclip_maximizer https://wiki.lesswrong.com/wiki/Paperclip_maximizer

goatlover 10y ago
The Paperclip maximizer assumes that you could have an AI smart enough to turn the world into paperclips, but not smart enough to understand what the human meant by "make me some paperclips". My guess is that paperclipping the world takes magnitudes more intelligence than understanding what a human means when saying something ambiguous.
- Smaug123 10y ago
  Not necessarily. It could be smart enough to know what you meant (and indeed to deduce the entire history of your thoughts!) and simply not care, because you told it to care about something else. For example, maybe you inadvertently programmed it to "be totally literally 100% certain that you've completed the task I told you to do". Then from its perspective, there's always a tiny, tiny chance that its sensors are being fooled or malfunctioning, so it can't be literally 100% certain that it's ever made a paperclip successfully, so by your programming, it should keep making more paperclips. This is independent of whether or not you wanted it to do that: it's a result of what you programmed it to do and what you told it to do.
  - goatlover 10y ago
    If a human being is smart enough to realize that you don't want to paperclip the entire world, then surely a super AI would be smart enough, by definition. Part of intelligence is knowing when to ignore all those tiny chances. Human intelligence is less brittle than artificial because we can handle ambiguity and know when it's ridiculous to continue a task, just because there might be some small chance that it's not finished. We also know that paperclipping the world will get in the way of other goals, like going to the show or making money.
    - Smaug123 10y ago
      I did indeed say that a superintelligent AI could realise that we don't want to paperclip the world. But it need not care what we want, unless we're really, really careful about how we program it. What is "ridiculous" about continuing a task because we're not certain that it's done yet? It's only your human moral system saying that. Just because humans usually don't value things enough to pursue them to the exclusion of all else…
      - goatlover 10y ago
        I tell my super intelligent assistant, who's much better at understanding human language than Siri, to make me some paperclips. It understands some to mean more than one and less than infinity. In fact, some means less than "a lot". The meaning of a lot depends on the context, which happens to be paperclips for me. What is "some" paperclips for me? It depends on how many papers I might need to clip (or whatever use I might have for paperclips). My super intelligent assistant would be able to work out a good estimate. After having an estimate, it can go make me "some" paperclips, and then stop somewhere short of paper clipping the entire world. Alternatively, it could just ask me how many "some" means.
        Hannan 10y ago
        - Hill Climbing - Local Maximum - Gradient Descent I don't pretend to have any sort of expertise in these sorts of discussions, so I thought I would throw out some easily wikipedia'd terms that seem to back your thoughts regarding convergence vs. exponentiation.
        Smaug123 10y ago
        You're still assuming that the assistant cares about doing what I mean, rather than doing precisely what I say or doing what I programmed it to do. That only happens if we're really, really careful about how we programmed the AI in the first place. I grant you 100% that the AI probably knows precisely how many paperclips I want, but you're assuming that it wants the same number of paperclips that I want. What an agent considers to be "good" is orthogonal to how intelligent that agent is. An agent of arbitrary intelligence can have arbitrary goals; the goals of an intelligent agent need not in principle look anything like those of a human. The only reason a superintelligent AI's goals would look like those of a human is because the humans very, very carefully programmed them into the AI somehow. Very, very careful programming is not a feature of how humans currently approach AI research.
- lazaroclapp 10y ago
  No. It means you could program an AI to a) make paperclips and b) increase its own intelligence. Assume you program (a) incorrectly (in the sense of lacking the appropriate safeguards about when not to make more paperclips) but (b) correctly. Then, it doesn't matter that the AI will eventually understand what you want perfectly, because what it will want to do is to keep making paperclips. The whole argument is that if you have a self-improving agent with a goal and the ability to improve in ways you can't predict at all, then you better make sure its goal matches yours. Of course, the obvious solution is: "make the objective function equivalent to 'do what I want you to do'", but the problem is we might not know how to encode that without help from a super-AI.
  - Smaug123 10y ago
    Even "do what I mean" is not obviously correct. Yudkowsky used to advocate the idea of "coherent extrapolated volition": the idea of what I would want the AI to do, were I much more intelligent, knew more, could think faster, were more how I wished I were, and so on.
    - lazaroclapp 10y ago
      "Do what I mean" is correct in that it reduces super-human AI to the same category of other human technologies: as good or as bad as whatever we choose to do with them. "Coherent extrapolated volition" might be preferable, but if we could get to the first, we could then ask for the second.
- deleted 10y ago
  [deleted]
- CM30 10y ago
  I don't think the intelligence aspect is the biggest problem with that paperclip maximiser concept. It's the physical ability to actually turn stuff into paperclips. Or whatever else the nearest analogy would be. Doesn't matter how determined the machine is to make them, the rest of the world won't exactly lay down and let it. Nor would its goals often be possible with the amount of resources available in its surroundings. It's basically the cat problems from this tongue in cheek article: http://idlewords.com/talks/superintelligence.htm http://idlewords.com/talks/superintelligence.htm
ffwd 10y ago
Yeah the problem is where would you put in the command to maximize paperclips? Even though deep learning is very powerful, it doesn't contain a utility function. The utility function to maximize paperclips would be some kind of module separate from a set of deep learning nets that would coordinate and read inputs from all the nets and then somehow give commands to some actuators somewhere (which it would also need intelligence about). All those deep learning networks would have to already be tuned to be very accurate in their own right before they are hooked up to any kind of controller. That was my point that, AI has been confused with the deep learning gains in the past years, even though, the hard part about how to coordinate all those inputs and also give the right commands to actuators is still not near to solved, much less a completely virtual superintelligence that has its own "virtual" goals and virtual simulations of both its inputs and potential actuators in the real world. This would be akin to a kind of intelligence algorithm, and not deep learning vector algorithms that reveal structures in data.

by jononor 10y ago

Check for instance the 'Paperclip maximizer' thought experiment, https://wiki.lesswrong.com/wiki/Paperclip_maximizer https://wiki.lesswrong.com/wiki/Paperclip_maximizer

goatlover 10y ago

The Paperclip maximizer assumes that you could have an AI smart enough to turn the world into paperclips, but not smart enough to understand what the human meant by "make me some paperclips". My guess is that paperclipping the world takes magnitudes more intelligence than understanding what a human means when saying something ambiguous.

Smaug123 10y ago
Not necessarily. It could be smart enough to know what you meant (and indeed to deduce the entire history of your thoughts!) and simply not care, because you told it to care about something else. For example, maybe you inadvertently programmed it to "be totally literally 100% certain that you've completed the task I told you to do". Then from its perspective, there's always a tiny, tiny chance that its sensors are being fooled or malfunctioning, so it can't be literally 100% certain that it's ever made a paperclip successfully, so by your programming, it should keep making more paperclips. This is independent of whether or not you wanted it to do that: it's a result of what you programmed it to do and what you told it to do.
- goatlover 10y ago
  If a human being is smart enough to realize that you don't want to paperclip the entire world, then surely a super AI would be smart enough, by definition. Part of intelligence is knowing when to ignore all those tiny chances. Human intelligence is less brittle than artificial because we can handle ambiguity and know when it's ridiculous to continue a task, just because there might be some small chance that it's not finished. We also know that paperclipping the world will get in the way of other goals, like going to the show or making money.
  - Smaug123 10y ago
    I did indeed say that a superintelligent AI could realise that we don't want to paperclip the world. But it need not care what we want, unless we're really, really careful about how we program it. What is "ridiculous" about continuing a task because we're not certain that it's done yet? It's only your human moral system saying that. Just because humans usually don't value things enough to pursue them to the exclusion of all else…
    - goatlover 10y ago
      I tell my super intelligent assistant, who's much better at understanding human language than Siri, to make me some paperclips. It understands some to mean more than one and less than infinity. In fact, some means less than "a lot". The meaning of a lot depends on the context, which happens to be paperclips for me. What is "some" paperclips for me? It depends on how many papers I might need to clip (or whatever use I might have for paperclips). My super intelligent assistant would be able to work out a good estimate. After having an estimate, it can go make me "some" paperclips, and then stop somewhere short of paper clipping the entire world. Alternatively, it could just ask me how many "some" means.
      - Hannan 10y ago
        - Hill Climbing - Local Maximum - Gradient Descent I don't pretend to have any sort of expertise in these sorts of discussions, so I thought I would throw out some easily wikipedia'd terms that seem to back your thoughts regarding convergence vs. exponentiation.
      - Smaug123 10y ago
        You're still assuming that the assistant cares about doing what I mean, rather than doing precisely what I say or doing what I programmed it to do. That only happens if we're really, really careful about how we programmed the AI in the first place. I grant you 100% that the AI probably knows precisely how many paperclips I want, but you're assuming that it wants the same number of paperclips that I want. What an agent considers to be "good" is orthogonal to how intelligent that agent is. An agent of arbitrary intelligence can have arbitrary goals; the goals of an intelligent agent need not in principle look anything like those of a human. The only reason a superintelligent AI's goals would look like those of a human is because the humans very, very carefully programmed them into the AI somehow. Very, very careful programming is not a feature of how humans currently approach AI research.
lazaroclapp 10y ago
No. It means you could program an AI to a) make paperclips and b) increase its own intelligence. Assume you program (a) incorrectly (in the sense of lacking the appropriate safeguards about when not to make more paperclips) but (b) correctly. Then, it doesn't matter that the AI will eventually understand what you want perfectly, because what it will want to do is to keep making paperclips. The whole argument is that if you have a self-improving agent with a goal and the ability to improve in ways you can't predict at all, then you better make sure its goal matches yours. Of course, the obvious solution is: "make the objective function equivalent to 'do what I want you to do'", but the problem is we might not know how to encode that without help from a super-AI.
- Smaug123 10y ago
  Even "do what I mean" is not obviously correct. Yudkowsky used to advocate the idea of "coherent extrapolated volition": the idea of what I would want the AI to do, were I much more intelligent, knew more, could think faster, were more how I wished I were, and so on.
  - lazaroclapp 10y ago
    "Do what I mean" is correct in that it reduces super-human AI to the same category of other human technologies: as good or as bad as whatever we choose to do with them. "Coherent extrapolated volition" might be preferable, but if we could get to the first, we could then ask for the second.
deleted 10y ago
[deleted]
CM30 10y ago
I don't think the intelligence aspect is the biggest problem with that paperclip maximiser concept. It's the physical ability to actually turn stuff into paperclips. Or whatever else the nearest analogy would be. Doesn't matter how determined the machine is to make them, the rest of the world won't exactly lay down and let it. Nor would its goals often be possible with the amount of resources available in its surroundings. It's basically the cat problems from this tongue in cheek article: http://idlewords.com/talks/superintelligence.htm http://idlewords.com/talks/superintelligence.htm

ffwd 10y ago

Yeah the problem is where would you put in the command to maximize paperclips? Even though deep learning is very powerful, it doesn't contain a utility function. The utility function to maximize paperclips would be some kind of module separate from a set of deep learning nets that would coordinate and read inputs from all the nets and then somehow give commands to some actuators somewhere (which it would also need intelligence about). All those deep learning networks would have to already be tuned to be very accurate in their own right before they are hooked up to any kind of controller. That was my point that, AI has been confused with the deep learning gains in the past years, even though, the hard part about how to coordinate all those inputs and also give the right commands to actuators is still not near to solved, much less a completely virtual superintelligence that has its own "virtual" goals and virtual simulations of both its inputs and potential actuators in the real world. This would be akin to a kind of intelligence algorithm, and not deep learning vector algorithms that reveal structures in data.