4 ms·

The issue is that it doesn’t have to be malicious to cause harm. That’s the point of the paper clip optimizer thought experiment. It’s not about motivation as m

The issue is that it doesn’t have to be malicious to cause harm. That’s the point of the paper clip optimizer thought experiment. It’s not about motivation as much as just what it happens to have learned to do.

bamboozled 4y ago
Here's the thing though, a paper clip optimizer would just be shut off, nothing "intelligent" is going to just make paper clips forever. Who or what will pay for the energy to do that? Like what are we describing here, a super intelligent being which just makes paper clips? Why would anything that could understand, think and or innovate do that?
- 6gvONxR4sf7o 4y ago
  Why would it not? Superintelligent doesn’t imply human-like intelligence. Prolog is superintelligent, for example. It may be more useful to say a super powerful being, rather than super intelligent, just to sidestep the anthropomorphizing. It’s easy to imagine a super powerful “stupid” (read: misaligned) bot going awry.
  - bamboozled 4y ago
    So we might somehow create something hyper intelligent/powerful we don't really understand or can't even fathom, but we're apparently going to create it anyway, and then some how program it to want to optimize for paper clips...or it might somehow just come to the conclusion that it should maximize the number of paper clips in the universe for some reason? All this AI / AGI speculation really sounds like we're inventing mythological like creatures, which is quite interesting when looking back through human history because they've always been part of the narrative at some stage only maybe this time, we might make something like that for real, or we're deluded and in 1000 years people will think we were primitive ancient thinkers, interesting times ahead.
    - 6gvONxR4sf7o 4y ago
      Close. We might create a very powerful system which we can't determine is fully aligned with our values, but can determine is at least loosely aligned with something we care about. Then we might direct that system to do something loosely aligned with our values (e.g. maximize paperclips), at which point it does something against our values as well. And it's not hypothetical. In RL, you do see models exploit simulation errors to maximize reward in a way we didn't intend and would have disallowed if we'd thought of it. Likewise across automation, we get what we built, not what we intended. We have solid historical precedent to expect to not get quite what we want, and the tech seems to be on the precipice of being very powerful, so I'd call it a very straightforward fear for us to worry about making something very powerful that isn't quite what we want.
      - bamboozled 4y ago
        Do you not think there isn't still a different between something which can become sentient, somehow start operating outside of the computer it lives in and on and on ? I mean it would have to be a pretty big leap from and algorithm optimized to do "something" to an algorithm which understands how to operate interdependently and then escape it's box and then start to do the wrong thing, and to add to that, be smart enough to stop us from turning it off before doing too much damage. Actually curious what people think about this?
        6gvONxR4sf7o 4y ago
        I think the whole idea of sentience is a red herring for reasoning about the potential for AI to cause harm. For example, take the title of the article we're talking about. You could take it with a dose of anthropomorphization and interpret it with human-like intent and sentience, or you could take it as an analogy that we might formally define closer to "not worried about AI that passes Turing test, but AI that fails for some reason other than the ability to succeed (and that other reason requires the evaluator to be misled)." So is sentience far away? No idea. Is misaligned AI a near-term worry? Yeah, absolutely. And "misaligned" doesn't have to be as extreme as 'makes humans extinct.' It can be as simple as 'is an asshole' or 'inexplicably swerves into the other lane 0.01% of the time' or something else we don't want in the system.
- naasking 4y ago
  > Here's the thing though, a paper clip optimizer would just be shut off, nothing "intelligent" is going to just make paper clips forever. Ostensibly, people are "intelligent". People act against their own rational long-term interests all the time. Point being, people have some kind of internal motivations that sometimes supersede rational choice. A paperclip optimizer, by definition, has the motivation to optimize the production of paperclips. This supersedes any other rational goals if they would interfere with making paperclips, and that's why it would be dangerous.

bamboozled 4y ago

Here's the thing though, a paper clip optimizer would just be shut off, nothing "intelligent" is going to just make paper clips forever. Who or what will pay for the energy to do that? Like what are we describing here, a super intelligent being which just makes paper clips? Why would anything that could understand, think and or innovate do that?

6gvONxR4sf7o 4y ago
Why would it not? Superintelligent doesn’t imply human-like intelligence. Prolog is superintelligent, for example. It may be more useful to say a super powerful being, rather than super intelligent, just to sidestep the anthropomorphizing. It’s easy to imagine a super powerful “stupid” (read: misaligned) bot going awry.
- bamboozled 4y ago
  So we might somehow create something hyper intelligent/powerful we don't really understand or can't even fathom, but we're apparently going to create it anyway, and then some how program it to want to optimize for paper clips...or it might somehow just come to the conclusion that it should maximize the number of paper clips in the universe for some reason? All this AI / AGI speculation really sounds like we're inventing mythological like creatures, which is quite interesting when looking back through human history because they've always been part of the narrative at some stage only maybe this time, we might make something like that for real, or we're deluded and in 1000 years people will think we were primitive ancient thinkers, interesting times ahead.
  - 6gvONxR4sf7o 4y ago
    Close. We might create a very powerful system which we can't determine is fully aligned with our values, but can determine is at least loosely aligned with something we care about. Then we might direct that system to do something loosely aligned with our values (e.g. maximize paperclips), at which point it does something against our values as well. And it's not hypothetical. In RL, you do see models exploit simulation errors to maximize reward in a way we didn't intend and would have disallowed if we'd thought of it. Likewise across automation, we get what we built, not what we intended. We have solid historical precedent to expect to not get quite what we want, and the tech seems to be on the precipice of being very powerful, so I'd call it a very straightforward fear for us to worry about making something very powerful that isn't quite what we want.
    - bamboozled 4y ago
      Do you not think there isn't still a different between something which can become sentient, somehow start operating outside of the computer it lives in and on and on ? I mean it would have to be a pretty big leap from and algorithm optimized to do "something" to an algorithm which understands how to operate interdependently and then escape it's box and then start to do the wrong thing, and to add to that, be smart enough to stop us from turning it off before doing too much damage. Actually curious what people think about this?
      - 6gvONxR4sf7o 4y ago
        I think the whole idea of sentience is a red herring for reasoning about the potential for AI to cause harm. For example, take the title of the article we're talking about. You could take it with a dose of anthropomorphization and interpret it with human-like intent and sentience, or you could take it as an analogy that we might formally define closer to "not worried about AI that passes Turing test, but AI that fails for some reason other than the ability to succeed (and that other reason requires the evaluator to be misled)." So is sentience far away? No idea. Is misaligned AI a near-term worry? Yeah, absolutely. And "misaligned" doesn't have to be as extreme as 'makes humans extinct.' It can be as simple as 'is an asshole' or 'inexplicably swerves into the other lane 0.01% of the time' or something else we don't want in the system.
naasking 4y ago
> Here's the thing though, a paper clip optimizer would just be shut off, nothing "intelligent" is going to just make paper clips forever. Ostensibly, people are "intelligent". People act against their own rational long-term interests all the time. Point being, people have some kind of internal motivations that sometimes supersede rational choice. A paperclip optimizer, by definition, has the motivation to optimize the production of paperclips. This supersedes any other rational goals if they would interfere with making paperclips, and that's why it would be dangerous.