3 ms·

The point is it’s virtually impossible to put constraints on it that make it do what you want because if it’s more intelligent than you it can always think of s

The point is it’s virtually impossible to put constraints on it that make it do what you want because if it’s more intelligent than you it can always think of something that you won’t, that’s technically within the rules you set but not at all intended. That’s why we’d need to make it care about the underlying intentions and values, but that’s also really hard

majormajor 3y ago
That seems like an attempt to set up a futile exercise in needle-threading that relies on narrow worst-case-scenario definitions of "superintelligence"/AGI. It's too intelligent to restrict our constraints. but It's not too intelligent to be "aligned" to underlying intentions and values? That approach doesn't even work on humans, why would it work on a superintelligence?
- nuancebydefault 3y ago
  > not too intelligent to be aligned The thing is, being aligned cannot be solved with intelligence per se. Say, you are (far) more intelligent than a spider. There's no way you can get aligned with (all of) its values unless the spider finds a way to let you know (all of) its values. Maybe the spider just tells you to make plenty of webs without knowing that it might get entangled in them by itself. The webs are analogous to the paperclips.
  - circuit10 3y ago
    It's less about not knowing the intentions and more that it has no reason to care about anything other than the goal you gave it
- circuit10 3y ago
  > It's not too intelligent to be "aligned" to underlying intentions and values? Intelligence makes that harder, not easier. Just because it can work out the underlying intentions doesn't mean it cares about them. Remember, this is an optimisation process that maximises a function; deciding to do something that doesn't maximise that function will not be selected for Whenever you do something, do you think "the underlying goal of my behaviour set by evolution is to reproduce and have children, so I'd better make sure my actions are aligned to the goal of doing that"? No, you don't care what the underlying "intentions" are, and neither does an AI. Because our environment has changed, many of our instincts no longer line up well with that goal, which is actually another problem with aligning AI because it can do the same thing if the environment changes since training as the training process can create an AI with goals that aren't exactly the same as the training goal but line up well during training
seventhtiger 3y ago
The basic premise is that it has somewhere in it that is telling it to make more paperclips. Put the constraints there. If you're saying such an AI would be too smart to be a simple paperclip maximizer, then I'd agree, but then what's the point of the thought experiment if a paperclip maximizer is impossible.
- travisjungroth 3y ago
  I think you’re missing some big pieces of the idea here. The first is that these constraints aren’t easy. Make paperclips in a way that doesn’t hurt anyone. Ok, so it’s going to make sure every single part is ethically sourced from a company that never causes any harm to come to anyone ever, and doesn’t give any money to people or companies that do? That doesn’t exist. So you put in a few caveats and those aren’t exactly easy to get right. The second part is an any versus all issue. Even if you get this right in any one case, that’s not enough. We have to get this right in all cases. So even if you can come up with an idea to make an ethical super intelligence, do you have an idea to make all super intelligences act ethically? I actually believe in the general premise of this question as being the biggest threat to humans. I don’t think it’s a doomsday bot that gets us. It’s going to be someone trying to hit a KPI, and they’ll make a super intelligence that demolishes us like a construction site over an anthill.
- circuit10 3y ago
  > The basic premise is that it has somewhere in it that is telling it to make more paperclips. Put the constraints there. What constraints do you suggest? If it's just changing "make as many paperclips as possible" to "make at least x number of paperclips" (putting a cap on the reward it gets), here's a good explanation of why that doesn't really work: https://www.youtube.com/watch?v=Ao4jwLwT36M https://www.youtube.com/watch?v=Ao4jwLwT36M If you're suggesting limiting the types of actions it can take, then to do that to the point that a superintelligence can't find a way around it (maybe letting it choose between one of two options and then shutting it down and never using it again) would make it not very useful, so you'd be better off just not making it at all > If you're saying such an AI would be too smart to be a simple paperclip maximizer No, that's not what I'm saying. Any goal is compatible with any level of intelligence, there is no reason why it wouldn't be possible to follow a simple goal in a complex way. Again here's a video about that: https://www.youtube.com/watch?v=hEUO6pjwFOo https://www.youtube.com/watch?v=hEUO6pjwFOo

majormajor 3y ago

That seems like an attempt to set up a futile exercise in needle-threading that relies on narrow worst-case-scenario definitions of "superintelligence"/AGI. It's too intelligent to restrict our constraints. but It's not too intelligent to be "aligned" to underlying intentions and values? That approach doesn't even work on humans, why would it work on a superintelligence?

nuancebydefault 3y ago
> not too intelligent to be aligned The thing is, being aligned cannot be solved with intelligence per se. Say, you are (far) more intelligent than a spider. There's no way you can get aligned with (all of) its values unless the spider finds a way to let you know (all of) its values. Maybe the spider just tells you to make plenty of webs without knowing that it might get entangled in them by itself. The webs are analogous to the paperclips.
- circuit10 3y ago
  It's less about not knowing the intentions and more that it has no reason to care about anything other than the goal you gave it
circuit10 3y ago
> It's not too intelligent to be "aligned" to underlying intentions and values? Intelligence makes that harder, not easier. Just because it can work out the underlying intentions doesn't mean it cares about them. Remember, this is an optimisation process that maximises a function; deciding to do something that doesn't maximise that function will not be selected for Whenever you do something, do you think "the underlying goal of my behaviour set by evolution is to reproduce and have children, so I'd better make sure my actions are aligned to the goal of doing that"? No, you don't care what the underlying "intentions" are, and neither does an AI. Because our environment has changed, many of our instincts no longer line up well with that goal, which is actually another problem with aligning AI because it can do the same thing if the environment changes since training as the training process can create an AI with goals that aren't exactly the same as the training goal but line up well during training

seventhtiger 3y ago

The basic premise is that it has somewhere in it that is telling it to make more paperclips. Put the constraints there. If you're saying such an AI would be too smart to be a simple paperclip maximizer, then I'd agree, but then what's the point of the thought experiment if a paperclip maximizer is impossible.

travisjungroth 3y ago
I think you’re missing some big pieces of the idea here. The first is that these constraints aren’t easy. Make paperclips in a way that doesn’t hurt anyone. Ok, so it’s going to make sure every single part is ethically sourced from a company that never causes any harm to come to anyone ever, and doesn’t give any money to people or companies that do? That doesn’t exist. So you put in a few caveats and those aren’t exactly easy to get right. The second part is an any versus all issue. Even if you get this right in any one case, that’s not enough. We have to get this right in all cases. So even if you can come up with an idea to make an ethical super intelligence, do you have an idea to make all super intelligences act ethically? I actually believe in the general premise of this question as being the biggest threat to humans. I don’t think it’s a doomsday bot that gets us. It’s going to be someone trying to hit a KPI, and they’ll make a super intelligence that demolishes us like a construction site over an anthill.
circuit10 3y ago
> The basic premise is that it has somewhere in it that is telling it to make more paperclips. Put the constraints there. What constraints do you suggest? If it's just changing "make as many paperclips as possible" to "make at least x number of paperclips" (putting a cap on the reward it gets), here's a good explanation of why that doesn't really work: https://www.youtube.com/watch?v=Ao4jwLwT36M https://www.youtube.com/watch?v=Ao4jwLwT36M If you're suggesting limiting the types of actions it can take, then to do that to the point that a superintelligence can't find a way around it (maybe letting it choose between one of two options and then shutting it down and never using it again) would make it not very useful, so you'd be better off just not making it at all > If you're saying such an AI would be too smart to be a simple paperclip maximizer No, that's not what I'm saying. Any goal is compatible with any level of intelligence, there is no reason why it wouldn't be possible to follow a simple goal in a complex way. Again here's a video about that: https://www.youtube.com/watch?v=hEUO6pjwFOo https://www.youtube.com/watch?v=hEUO6pjwFOo