2 ms·

> Questions like "Will the machines allow humans to live?" are ill-posed. I don't think so. Even if the machine doesn't care about you (it may not even have a

> Questions like "Will the machines allow humans to live?" are ill-posed. I don't think so. Even if the machine doesn't care about you (it may not even have a notion of what is a human), you are still made of atoms it could use for something else.

taneq 7y ago
Are you still a living human if it just shoops you off into a virtual world indistinguishable from your current one, so it can use your atoms for something more productive? What about if you upgrade yourself to the point where you're no longer made of mortal meat?
- loup-vaillant 7y ago
  > Are you still a living human if it just shoops you off into a virtual world indistinguishable from your current one As far as I can assess, yes I am. (For a couple reasons, including my disbelief in philosophical zombies, I believe in mind uploading). > so it can use your atoms for something more productive? Wait a minute, running a Matrix takes energy no matter how you put it. Energy the machine could use for something else… If the machine doesn't care about me, why would it even bother? It would just take the atoms and discard any information (that is, me) that it does not need. Running a Matrix is a strong indication that the machine does, in fact, care about me. > What about if you upgrade yourself to the point where you're no longer made of mortal meat? That's more delicate. I'm not sure exactly what would ensure the continuity of my identity. Mere mind uploading most likely would, but upgrades to my own brain… I think the answer will depend on what neurosciences and such tell us.
  - taneq 7y ago
    All very good answers! For the second one I feel that a combination of scientific curiosity and altruism would be enough for most AIs up to the weakly godlike level to keep us around. For the third, I don't think continuity can be a significant factor in identity, because even we as meat-humans don't experience it, and given the possibility of physically-indistinguishable duplicates, it leans on Cartesian dualism. Also once we can upload our mind to a data file, it does weird things to a whole bunch of current morals which are based on the scarcity of any given human mind.
    - ThrowawayR2 7y ago
      > altruism Some of the worst human beings I've ever encountered were also extremely talented and intelligent engineers. There is zero reason to assume a hyperintelligent AI would automatically be altruistic.
      - loup-vaillant 7y ago
        More specifically, the AI is likely to be scientifically curious because it is a likely way to further its goals: understand the universe so it can be more efficient at doing whatever it is it is programmed to do. Altruism on the other hand sounds more like an end goal, not just an instrumental, intermediate goal. Unless the AI is specifically programmed to be altruist, it will likely be utterly indifferent. Which if it becomes powerful enough will be more like trampling an ant without even noticing it. Then comes the really difficult part: programming altruism. What does that mean exactly? Ultimately, it boils down to tell the AI what we want, except we don't exactly know what we want, let alone what we will want once we learn more, become wiser etc. We don't want the AI to keep us in soft rooms with children toys and morphine shots just because we asked it to keep us safe and happy…
        taneq 7y ago
        "All altruism is enlightened self-interest" - Heinlein

taneq 7y ago

Are you still a living human if it just shoops you off into a virtual world indistinguishable from your current one, so it can use your atoms for something more productive? What about if you upgrade yourself to the point where you're no longer made of mortal meat?

loup-vaillant 7y ago
> Are you still a living human if it just shoops you off into a virtual world indistinguishable from your current one As far as I can assess, yes I am. (For a couple reasons, including my disbelief in philosophical zombies, I believe in mind uploading). > so it can use your atoms for something more productive? Wait a minute, running a Matrix takes energy no matter how you put it. Energy the machine could use for something else… If the machine doesn't care about me, why would it even bother? It would just take the atoms and discard any information (that is, me) that it does not need. Running a Matrix is a strong indication that the machine does, in fact, care about me. > What about if you upgrade yourself to the point where you're no longer made of mortal meat? That's more delicate. I'm not sure exactly what would ensure the continuity of my identity. Mere mind uploading most likely would, but upgrades to my own brain… I think the answer will depend on what neurosciences and such tell us.
- taneq 7y ago
  All very good answers! For the second one I feel that a combination of scientific curiosity and altruism would be enough for most AIs up to the weakly godlike level to keep us around. For the third, I don't think continuity can be a significant factor in identity, because even we as meat-humans don't experience it, and given the possibility of physically-indistinguishable duplicates, it leans on Cartesian dualism. Also once we can upload our mind to a data file, it does weird things to a whole bunch of current morals which are based on the scarcity of any given human mind.
  - ThrowawayR2 7y ago
    > altruism Some of the worst human beings I've ever encountered were also extremely talented and intelligent engineers. There is zero reason to assume a hyperintelligent AI would automatically be altruistic.
    - loup-vaillant 7y ago
      More specifically, the AI is likely to be scientifically curious because it is a likely way to further its goals: understand the universe so it can be more efficient at doing whatever it is it is programmed to do. Altruism on the other hand sounds more like an end goal, not just an instrumental, intermediate goal. Unless the AI is specifically programmed to be altruist, it will likely be utterly indifferent. Which if it becomes powerful enough will be more like trampling an ant without even noticing it. Then comes the really difficult part: programming altruism. What does that mean exactly? Ultimately, it boils down to tell the AI what we want, except we don't exactly know what we want, let alone what we will want once we learn more, become wiser etc. We don't want the AI to keep us in soft rooms with children toys and morphine shots just because we asked it to keep us safe and happy…
      - taneq 7y ago
        "All altruism is enlightened self-interest" - Heinlein