4 ms·

"The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else." http://singinst.org/upload/artificial-inte

"The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else." http://singinst.org/upload/artificial-intelligence-risk.pdf http://singinst.org/upload/artificial-intelligence-risk.pdf

zeteo 16y ago
And the AI also can use you for something else, or leave you alone altogether and follow other pursuits. It's also theoretically possible that some intelligent person you just met is actually dr. Hannibal Lecter and would like to use your constituent atoms for his culinary delight. But you don't worry much about that, do you?
- jerf 16y ago
  The most common failure case in thinking about AI by far is insisting on putting them in a human context. Whatever else they may be, they will not be human. They will not come with the heritage billions of years of biological evolution, with many of the past several million being spent in increasingly cooperative scenarios which has endowed us with social behaviors so deeply ingrained in our very genes we can hardly conceive of a being not having them. Even our pathological humans like Lecter are still far, far more human than a random AI will be. Hopefully we'll give them something else that will put them on a similar moral footing, but it won't happen automatically, and they still won't be human after that.
  - zeteo 16y ago
    The evolutionary baggage is most definitely not one of cooperation, at least not beyond our immediate "monkeysphere"; millennia of wars and massacres bear witness to that. The primary reason we usually avoid violence is not that we find it irresistibly repugnant, but because we've become intelligent enough to realize it normally creates more problems than it solves. AI not being human (for some definition of human) also means it doesn't have to be jealous, hateful and short-sighted.
    - TheEzEzz 16y ago
      The primary reason we usually avoid violence is not that we find it irresistibly repugnant, but because we've become intelligent enough to realize it normally creates more problems than it solves. Do you honestly believe that most people act morally purely on consequentialist grounds? I suspect the vast majority of people wouldn't kill someone even if it benefited them and they knew they would never get caught. There are human sociopaths however, and it is the case that if they were more powerful than me I would be very scared. A very intelligent AI would avoid conflict so long as it thought it would be to its detriment. If it was confident that it was powerful enough to take what it wanted by force without losing much in the process, it would do so.
      - zeteo 16y ago
        > I suspect the vast majority of people wouldn't kill someone even if it benefited them and they knew they would never get caught. I wasn't referring to individualized, but rather to large-scale violence. And, having read a bit about history, I must disagree. The extreme enthusiasm that is sometimes displayed in regards to wars (e.g., in the initial stages of the US Civil War and of WW1) shows that we are more than capable of wishing, and striving for, the death of out-group members. The main reason that large-scale wars have been avoided for the last 65 years has been exactly consequentialist: the wiser among us have learned the bitter lessons of our bloody past. As the Milgram experiment (not to talk about the experience of conscript mass armies) has shown, the vast majority of people are more than capable of killing someone.
      - erikpukinskis 16y ago
        A very intelligent AI would avoid conflict so long as it thought it would be to its detriment. Most intelligent beings don't actually "think" about their detriment. Consider the airplane+pilot combination. They don't actually "think" about their specific actions and all the possible terrible outcomes and weigh them. They have simply evolved checklists over time that lead to incredibly smart, adaptive behaviors. I have no reason to think robots will be any different. Surely they will develop patterns that are highly adaptive, but I doubt they will be entirely, or even primarily based on any kind of thinking or analysis. They will be based on experience, as situated intelligence often is, and therefore susceptible to similar mistakes that humans make. And to answer your original question, humans don't necessarily think about consequences, but surely 1000s of years of consequences are built into our cultural practices, and humans act almost entirely based on rough-and-ready, contextually assembled cultural practices.
- bermanoid 16y ago
  And the AI also can use you for something else, or leave you alone altogether and follow other pursuits. Right. But the problem is, an exponentially growing AI that's following other pursuits without worrying about us will, with almost 100% certainty, kill us off by accident as it expands. We really need to make sure it does care about us, and that all future generations will care about us, including only building future designs that care about us, etc., and making all of these constraints close to 100% resistant to accidental bit flips, errors, etc. You'd worry a lot more about a potential Hannibal Lecter maybe wanting to eat your brains if he was not 3x smarter than you, but 1,000x smarter, and doubling his intelligence every twenty minutes. Whether he wants to do it now or not is not the issue; the issue is that every twenty minutes, he's a completely different entity whose goals may not match up at all with what they were before, unless the design is so careful and brilliant that certain constraints reliably propagate forward.
icey 16y ago
I don't get the insistence that AI would be content staying Earthbound. If it needs atoms, why not leave Earth? Machines don't need food & water, they need fuel. The universe has lots of that.
- TheEzEzz 16y ago
  Earthbound resources are drastically cheaper.
  - weavejester 16y ago
    How so? Sure, there's a significant upfront cost to sending a rocket out to the asteroid belt, but you only have to do that once. Once there's a base with a radio antenna, machine intelligences can travel back and forth at the speed of light. And once you're in the asteroid belt, you have a huge amount of unclaimed raw material, unlimited space, and no gravity well to get in the way.
- barry-cotter 16y ago
  Yes, but until such time as it gets off the earth is the most convenient source of atoms, and being that you're on the surface at the bottom of a big gravity well you're pretty convenient.

zeteo 16y ago

And the AI also can use you for something else, or leave you alone altogether and follow other pursuits. It's also theoretically possible that some intelligent person you just met is actually dr. Hannibal Lecter and would like to use your constituent atoms for his culinary delight. But you don't worry much about that, do you?

jerf 16y ago
The most common failure case in thinking about AI by far is insisting on putting them in a human context. Whatever else they may be, they will not be human. They will not come with the heritage billions of years of biological evolution, with many of the past several million being spent in increasingly cooperative scenarios which has endowed us with social behaviors so deeply ingrained in our very genes we can hardly conceive of a being not having them. Even our pathological humans like Lecter are still far, far more human than a random AI will be. Hopefully we'll give them something else that will put them on a similar moral footing, but it won't happen automatically, and they still won't be human after that.
- zeteo 16y ago
  The evolutionary baggage is most definitely not one of cooperation, at least not beyond our immediate "monkeysphere"; millennia of wars and massacres bear witness to that. The primary reason we usually avoid violence is not that we find it irresistibly repugnant, but because we've become intelligent enough to realize it normally creates more problems than it solves. AI not being human (for some definition of human) also means it doesn't have to be jealous, hateful and short-sighted.
  - TheEzEzz 16y ago
    The primary reason we usually avoid violence is not that we find it irresistibly repugnant, but because we've become intelligent enough to realize it normally creates more problems than it solves. Do you honestly believe that most people act morally purely on consequentialist grounds? I suspect the vast majority of people wouldn't kill someone even if it benefited them and they knew they would never get caught. There are human sociopaths however, and it is the case that if they were more powerful than me I would be very scared. A very intelligent AI would avoid conflict so long as it thought it would be to its detriment. If it was confident that it was powerful enough to take what it wanted by force without losing much in the process, it would do so.
    - zeteo 16y ago
      > I suspect the vast majority of people wouldn't kill someone even if it benefited them and they knew they would never get caught. I wasn't referring to individualized, but rather to large-scale violence. And, having read a bit about history, I must disagree. The extreme enthusiasm that is sometimes displayed in regards to wars (e.g., in the initial stages of the US Civil War and of WW1) shows that we are more than capable of wishing, and striving for, the death of out-group members. The main reason that large-scale wars have been avoided for the last 65 years has been exactly consequentialist: the wiser among us have learned the bitter lessons of our bloody past. As the Milgram experiment (not to talk about the experience of conscript mass armies) has shown, the vast majority of people are more than capable of killing someone.
    - erikpukinskis 16y ago
      A very intelligent AI would avoid conflict so long as it thought it would be to its detriment. Most intelligent beings don't actually "think" about their detriment. Consider the airplane+pilot combination. They don't actually "think" about their specific actions and all the possible terrible outcomes and weigh them. They have simply evolved checklists over time that lead to incredibly smart, adaptive behaviors. I have no reason to think robots will be any different. Surely they will develop patterns that are highly adaptive, but I doubt they will be entirely, or even primarily based on any kind of thinking or analysis. They will be based on experience, as situated intelligence often is, and therefore susceptible to similar mistakes that humans make. And to answer your original question, humans don't necessarily think about consequences, but surely 1000s of years of consequences are built into our cultural practices, and humans act almost entirely based on rough-and-ready, contextually assembled cultural practices.
bermanoid 16y ago
And the AI also can use you for something else, or leave you alone altogether and follow other pursuits. Right. But the problem is, an exponentially growing AI that's following other pursuits without worrying about us will, with almost 100% certainty, kill us off by accident as it expands. We really need to make sure it does care about us, and that all future generations will care about us, including only building future designs that care about us, etc., and making all of these constraints close to 100% resistant to accidental bit flips, errors, etc. You'd worry a lot more about a potential Hannibal Lecter maybe wanting to eat your brains if he was not 3x smarter than you, but 1,000x smarter, and doubling his intelligence every twenty minutes. Whether he wants to do it now or not is not the issue; the issue is that every twenty minutes, he's a completely different entity whose goals may not match up at all with what they were before, unless the design is so careful and brilliant that certain constraints reliably propagate forward.

icey 16y ago

I don't get the insistence that AI would be content staying Earthbound. If it needs atoms, why not leave Earth? Machines don't need food & water, they need fuel. The universe has lots of that.

TheEzEzz 16y ago
Earthbound resources are drastically cheaper.
- weavejester 16y ago
  How so? Sure, there's a significant upfront cost to sending a rocket out to the asteroid belt, but you only have to do that once. Once there's a base with a radio antenna, machine intelligences can travel back and forth at the speed of light. And once you're in the asteroid belt, you have a huge amount of unclaimed raw material, unlimited space, and no gravity well to get in the way.
barry-cotter 16y ago
Yes, but until such time as it gets off the earth is the most convenient source of atoms, and being that you're on the surface at the bottom of a big gravity well you're pretty convenient.