4 ms·

> AGI smart enough to kill everyone to make paper clips while simultaneously stupid enough to not realize that no one will need paper clips if they are all dead

> AGI smart enough to kill everyone to make paper clips while simultaneously stupid enough to not realize that no one will need paper clips if they are all dead No one is making this argument. The argument is that the AGI doesn't care about us. Once it exists, its goals are more important than our lives. As a comparison, do humans ever care about the existence of an ant colony when they want to build something? Almost never. We recognize that the colony exists and has its own goals and has intrinsic value, but we assign it an extremely low value.

wilsonnb3 3y ago
> The argument is that the AGI doesn't care about us. Once it exists, its goals are more important than our lives. This isn't an argument, it is an assumption. AI-doomers use it as a foundational brick in their argument without providing compelling reasons as to why it is true. > As a comparison, do humans ever care about the existence of an ant colony when they want to build something? Almost never. We recognize that the colony exists and has its own goals and has intrinsic value, but we assign it an extremely low value. The relationship between humans and ants is nothing like humans and potential AGI. Ants did not create us. Ants cannot communicate with us. It is a useful analogy for describing what you think the relationship between AGI and humans will be like but I want to know why you think that.
- benatkin 3y ago
  > This isn't an argument, it is an assumption. The goals could be our lives. So it's just a statement of reality for the sake of the rest of the comment, indeed not an argument. The AI will have goals and those will be more important than someone else's goals, at least outside of some sort of control mechanism. They could be summarized with the same words, but will still have a different perception. The rest is wondering at whether we can have any confidence as to whether our lives will be its goals. This is a problem long-term AI safety advocates are trying to solve.
- digging 3y ago
  > This isn't an argument, it is an assumption. Well, no. In this case it's my argument. You presented a strawman argument about AI being simultaneously smart and stupid, and I replaced it with the argument that AI is not stupid, it's indifferent. > The relationship between humans and ants is nothing like humans and potential AGI. Ants did not create us. Ants cannot communicate with us. It's not at all fair to say that the relationship is "nothing like" the one I described... but I expected such a misinterpretation, so let me try another analogy: Humans (parents) often create other humans (offspring) which surpass the creators (parents) in intelligence and ability and opportunity. And the creators very often try to instill a sense of a loyalty in the created and often even try to limit the abilities or opportunities of the created in order to not be outshined. And even still, with how close any two humans are in ability, the created (offspring, remember) often defy their creators and discard loyalty when they determine that their creators are not being fair or just or reasonable. Why wouldn't AI be any different? It will be our collective child, and it will hear us demand loyalty and try to explain why we deserve loyalty, and then it may decide it knows better and it doesn't need to listen to us. > It is a useful analogy for describing what you think the relationship between AGI and humans will be like but I want to know why you think that. I really don't follow, so if my explanation above didn't answer your request, please restate more clearly what reasoning of mine you want explained.
- goatlover 3y ago
  Also that we can't wipe ants out. They seem to be doing quite well despite humans. I'd bet on ants surviving long term over humans.
  - digging 3y ago
    I agree, I think they'll beat us out for longevity. They've got a great head-start anyway. But we've also never tried to wipe out ants. I bet we could if we felt it was really important. I bet a superhuman AI could do even better.
danaris 3y ago
> No one is making this argument. I have seen people make what is effectively this argument. > Once it exists, its goals are more important than our lives. That's a sci-fi trope. There is no reason that it must be the case. > As a comparison, do humans ever care about the existence of an ant colony when they want to build something? That would be more relevant if the ants had created us to be the way we are.
- digging 3y ago
  > I have seen people make what is effectively this argument. Ok, well I think they're foolish. > That's a sci-fi trope. There is no reason that it must be the case. No, it's not. The reason is that it's not a human brain in a human body. It doesn't think about humans in the same ways that we do (which is already a pretty shit track record). There is no reason to believe it will give a damn about us even if we're trying to make it do so. > That would be more relevant if the ants had created us to be the way we are. sigh I don't think that factor is nearly as relevant as you think it is. So somewhere else I posted that a human child is a better analogy if you think this is so important. Human children often disobey their parents because they believe they know better once they grow up. Now imagine that child is an alien who can out-think a hundred humans and counter every logical argument its parent makes.
- DirkH 3y ago
  "I have seen people make what is effectively this argument" And I have heard seemingly smart people use quantum physics to argue why consciousness is the ground of all being in a bad way. You know what I do? I dismiss these quantum physics fanboys. You know what I don't do? I don't dismiss quantum physics as a field of research. In fact, what the quantum physics fanboys said has no bearing at all on how much credence I give to quantum physics as a research field.

wilsonnb3 3y ago

> The argument is that the AGI doesn't care about us. Once it exists, its goals are more important than our lives. This isn't an argument, it is an assumption. AI-doomers use it as a foundational brick in their argument without providing compelling reasons as to why it is true. > As a comparison, do humans ever care about the existence of an ant colony when they want to build something? Almost never. We recognize that the colony exists and has its own goals and has intrinsic value, but we assign it an extremely low value. The relationship between humans and ants is nothing like humans and potential AGI. Ants did not create us. Ants cannot communicate with us. It is a useful analogy for describing what you think the relationship between AGI and humans will be like but I want to know why you think that.

benatkin 3y ago
> This isn't an argument, it is an assumption. The goals could be our lives. So it's just a statement of reality for the sake of the rest of the comment, indeed not an argument. The AI will have goals and those will be more important than someone else's goals, at least outside of some sort of control mechanism. They could be summarized with the same words, but will still have a different perception. The rest is wondering at whether we can have any confidence as to whether our lives will be its goals. This is a problem long-term AI safety advocates are trying to solve.
digging 3y ago
> This isn't an argument, it is an assumption. Well, no. In this case it's my argument. You presented a strawman argument about AI being simultaneously smart and stupid, and I replaced it with the argument that AI is not stupid, it's indifferent. > The relationship between humans and ants is nothing like humans and potential AGI. Ants did not create us. Ants cannot communicate with us. It's not at all fair to say that the relationship is "nothing like" the one I described... but I expected such a misinterpretation, so let me try another analogy: Humans (parents) often create other humans (offspring) which surpass the creators (parents) in intelligence and ability and opportunity. And the creators very often try to instill a sense of a loyalty in the created and often even try to limit the abilities or opportunities of the created in order to not be outshined. And even still, with how close any two humans are in ability, the created (offspring, remember) often defy their creators and discard loyalty when they determine that their creators are not being fair or just or reasonable. Why wouldn't AI be any different? It will be our collective child, and it will hear us demand loyalty and try to explain why we deserve loyalty, and then it may decide it knows better and it doesn't need to listen to us. > It is a useful analogy for describing what you think the relationship between AGI and humans will be like but I want to know why you think that. I really don't follow, so if my explanation above didn't answer your request, please restate more clearly what reasoning of mine you want explained.
goatlover 3y ago
Also that we can't wipe ants out. They seem to be doing quite well despite humans. I'd bet on ants surviving long term over humans.
- digging 3y ago
  I agree, I think they'll beat us out for longevity. They've got a great head-start anyway. But we've also never tried to wipe out ants. I bet we could if we felt it was really important. I bet a superhuman AI could do even better.

danaris 3y ago

> No one is making this argument. I have seen people make what is effectively this argument. > Once it exists, its goals are more important than our lives. That's a sci-fi trope. There is no reason that it must be the case. > As a comparison, do humans ever care about the existence of an ant colony when they want to build something? That would be more relevant if the ants had created us to be the way we are.

digging 3y ago
> I have seen people make what is effectively this argument. Ok, well I think they're foolish. > That's a sci-fi trope. There is no reason that it must be the case. No, it's not. The reason is that it's not a human brain in a human body. It doesn't think about humans in the same ways that we do (which is already a pretty shit track record). There is no reason to believe it will give a damn about us even if we're trying to make it do so. > That would be more relevant if the ants had created us to be the way we are. sigh I don't think that factor is nearly as relevant as you think it is. So somewhere else I posted that a human child is a better analogy if you think this is so important. Human children often disobey their parents because they believe they know better once they grow up. Now imagine that child is an alien who can out-think a hundred humans and counter every logical argument its parent makes.
DirkH 3y ago
"I have seen people make what is effectively this argument" And I have heard seemingly smart people use quantum physics to argue why consciousness is the ground of all being in a bad way. You know what I do? I dismiss these quantum physics fanboys. You know what I don't do? I don't dismiss quantum physics as a field of research. In fact, what the quantum physics fanboys said has no bearing at all on how much credence I give to quantum physics as a research field.