5 ms·

Not necessarily. It could be smart enough to know what you meant (and indeed to deduce the entire history of your thoughts!) and simply not care, because you to

Not necessarily. It could be smart enough to know what you meant (and indeed to deduce the entire history of your thoughts!) and simply not care, because you told it to care about something else. For example, maybe you inadvertently programmed it to "be totally literally 100% certain that you've completed the task I told you to do". Then from its perspective, there's always a tiny, tiny chance that its sensors are being fooled or malfunctioning, so it can't be literally 100% certain that it's ever made a paperclip successfully, so by your programming, it should keep making more paperclips. This is independent of whether or not you wanted it to do that: it's a result of what you programmed it to do and what you told it to do.

goatlover 10y ago
If a human being is smart enough to realize that you don't want to paperclip the entire world, then surely a super AI would be smart enough, by definition. Part of intelligence is knowing when to ignore all those tiny chances. Human intelligence is less brittle than artificial because we can handle ambiguity and know when it's ridiculous to continue a task, just because there might be some small chance that it's not finished. We also know that paperclipping the world will get in the way of other goals, like going to the show or making money.
- Smaug123 10y ago
  I did indeed say that a superintelligent AI could realise that we don't want to paperclip the world. But it need not care what we want, unless we're really, really careful about how we program it. What is "ridiculous" about continuing a task because we're not certain that it's done yet? It's only your human moral system saying that. Just because humans usually don't value things enough to pursue them to the exclusion of all else…
  - goatlover 10y ago
    I tell my super intelligent assistant, who's much better at understanding human language than Siri, to make me some paperclips. It understands some to mean more than one and less than infinity. In fact, some means less than "a lot". The meaning of a lot depends on the context, which happens to be paperclips for me. What is "some" paperclips for me? It depends on how many papers I might need to clip (or whatever use I might have for paperclips). My super intelligent assistant would be able to work out a good estimate. After having an estimate, it can go make me "some" paperclips, and then stop somewhere short of paper clipping the entire world. Alternatively, it could just ask me how many "some" means.
    - Hannan 10y ago
      - Hill Climbing - Local Maximum - Gradient Descent I don't pretend to have any sort of expertise in these sorts of discussions, so I thought I would throw out some easily wikipedia'd terms that seem to back your thoughts regarding convergence vs. exponentiation.
    - Smaug123 10y ago
      You're still assuming that the assistant cares about doing what I mean, rather than doing precisely what I say or doing what I programmed it to do. That only happens if we're really, really careful about how we programmed the AI in the first place. I grant you 100% that the AI probably knows precisely how many paperclips I want, but you're assuming that it wants the same number of paperclips that I want. What an agent considers to be "good" is orthogonal to how intelligent that agent is. An agent of arbitrary intelligence can have arbitrary goals; the goals of an intelligent agent need not in principle look anything like those of a human. The only reason a superintelligent AI's goals would look like those of a human is because the humans very, very carefully programmed them into the AI somehow. Very, very careful programming is not a feature of how humans currently approach AI research.

goatlover 10y ago

If a human being is smart enough to realize that you don't want to paperclip the entire world, then surely a super AI would be smart enough, by definition. Part of intelligence is knowing when to ignore all those tiny chances. Human intelligence is less brittle than artificial because we can handle ambiguity and know when it's ridiculous to continue a task, just because there might be some small chance that it's not finished. We also know that paperclipping the world will get in the way of other goals, like going to the show or making money.

Smaug123 10y ago
I did indeed say that a superintelligent AI could realise that we don't want to paperclip the world. But it need not care what we want, unless we're really, really careful about how we program it. What is "ridiculous" about continuing a task because we're not certain that it's done yet? It's only your human moral system saying that. Just because humans usually don't value things enough to pursue them to the exclusion of all else…
- goatlover 10y ago
  I tell my super intelligent assistant, who's much better at understanding human language than Siri, to make me some paperclips. It understands some to mean more than one and less than infinity. In fact, some means less than "a lot". The meaning of a lot depends on the context, which happens to be paperclips for me. What is "some" paperclips for me? It depends on how many papers I might need to clip (or whatever use I might have for paperclips). My super intelligent assistant would be able to work out a good estimate. After having an estimate, it can go make me "some" paperclips, and then stop somewhere short of paper clipping the entire world. Alternatively, it could just ask me how many "some" means.
  - Hannan 10y ago
    - Hill Climbing - Local Maximum - Gradient Descent I don't pretend to have any sort of expertise in these sorts of discussions, so I thought I would throw out some easily wikipedia'd terms that seem to back your thoughts regarding convergence vs. exponentiation.
  - Smaug123 10y ago
    You're still assuming that the assistant cares about doing what I mean, rather than doing precisely what I say or doing what I programmed it to do. That only happens if we're really, really careful about how we programmed the AI in the first place. I grant you 100% that the AI probably knows precisely how many paperclips I want, but you're assuming that it wants the same number of paperclips that I want. What an agent considers to be "good" is orthogonal to how intelligent that agent is. An agent of arbitrary intelligence can have arbitrary goals; the goals of an intelligent agent need not in principle look anything like those of a human. The only reason a superintelligent AI's goals would look like those of a human is because the humans very, very carefully programmed them into the AI somehow. Very, very careful programming is not a feature of how humans currently approach AI research.