5 ms·

I really like Vernor Vinge's fiction, and he's got some really big ideas, but sometimes in contexts like these I'm surprised by how shallow and provincial the t

by JackFr 7y ago

I really like Vernor Vinge's fiction, and he's got some really big ideas, but sometimes in contexts like these I'm surprised by how shallow and provincial the thinking is. He seems unable to step out the present western 20th/21st century worldview. The singularity (if it comes) should feature intelligences whose modes and motivations are so superior and advanced, they are literally incomprehensible to us. Questions like "Will the machines allow humans to live?" are ill-posed. As human philosophers can't agree on the meaning or the existence of free will, will the machines? Will they have will, free or otherwise? Or is free will itself the mere shadow on the wall of the a richer thing that we can't truly, ever comprehend? So talking about what the machines/intelligences 'want' is getting ahead of ourselves.

DennisP 7y ago
Possibly Vinge is well aware that a superior intelligence would be incomprehensible, but he just hasn't figured out how to write a compelling story about that.
tlb 7y ago
The question might be impossible to answer until after the fact, but it's not ill-posed. Machines either will, or will not, kill all humans.
pessimizer 7y ago
No, AI will desperately want sex, money, and lebensraum, just like human dictators. It won't think of either humanity or self-preservation as completely irrelevant and not interesting. Singularity talk is just projection and self-idealization. "AI will be like me, but perfect, and infinitely more capable of doing what I would do if I were omnipotent, which it will decide to do faster than I could ever make a decision."
loup-vaillant 7y ago
> Questions like "Will the machines allow humans to live?" are ill-posed. I don't think so. Even if the machine doesn't care about you (it may not even have a notion of what is a human), you are still made of atoms it could use for something else.
- taneq 7y ago
  Are you still a living human if it just shoops you off into a virtual world indistinguishable from your current one, so it can use your atoms for something more productive? What about if you upgrade yourself to the point where you're no longer made of mortal meat?
  - loup-vaillant 7y ago
    > Are you still a living human if it just shoops you off into a virtual world indistinguishable from your current one As far as I can assess, yes I am. (For a couple reasons, including my disbelief in philosophical zombies, I believe in mind uploading). > so it can use your atoms for something more productive? Wait a minute, running a Matrix takes energy no matter how you put it. Energy the machine could use for something else… If the machine doesn't care about me, why would it even bother? It would just take the atoms and discard any information (that is, me) that it does not need. Running a Matrix is a strong indication that the machine does, in fact, care about me. > What about if you upgrade yourself to the point where you're no longer made of mortal meat? That's more delicate. I'm not sure exactly what would ensure the continuity of my identity. Mere mind uploading most likely would, but upgrades to my own brain… I think the answer will depend on what neurosciences and such tell us.
    - taneq 7y ago
      All very good answers! For the second one I feel that a combination of scientific curiosity and altruism would be enough for most AIs up to the weakly godlike level to keep us around. For the third, I don't think continuity can be a significant factor in identity, because even we as meat-humans don't experience it, and given the possibility of physically-indistinguishable duplicates, it leans on Cartesian dualism. Also once we can upload our mind to a data file, it does weird things to a whole bunch of current morals which are based on the scarcity of any given human mind.
      - ThrowawayR2 7y ago
        > altruism Some of the worst human beings I've ever encountered were also extremely talented and intelligent engineers. There is zero reason to assume a hyperintelligent AI would automatically be altruistic.
skissane 7y ago
I think part of the problem with the question is that intelligence and motivations/drives are orthogonal. Intelligence is just a means to an end, it says nothing about the end to which it is a means. To say that a being is vastly more intelligent than human beings (super-intelligent) says nothing about what motivations it may have. Our own motivations are mostly given to us by our genetic and cultural inheritance, rather than something we've constructed for ourselves by our own intelligence. I think it likely, that any super-intelligent AI created by human beings, will be given motivations which somehow reflect the motivations and values of its creators. Not necessarily exactly the same – a silicon-based life-form has no need for food or sex as carbon-based life-forms do – but, I think a super-intelligent AI created by humans will likely be given motivations to seek to favour the interests of its creators, whether "its creators" are construed broadly (all of humanity as a whole) or narrowly (just the individual or corporation or nation-state which creates it.)

by JackFr 7y ago

DennisP 7y ago

Possibly Vinge is well aware that a superior intelligence would be incomprehensible, but he just hasn't figured out how to write a compelling story about that.

tlb 7y ago

The question might be impossible to answer until after the fact, but it's not ill-posed. Machines either will, or will not, kill all humans.

pessimizer 7y ago

No, AI will desperately want sex, money, and lebensraum, just like human dictators. It won't think of either humanity or self-preservation as completely irrelevant and not interesting. Singularity talk is just projection and self-idealization. "AI will be like me, but perfect, and infinitely more capable of doing what I would do if I were omnipotent, which it will decide to do faster than I could ever make a decision."

loup-vaillant 7y ago

> Questions like "Will the machines allow humans to live?" are ill-posed. I don't think so. Even if the machine doesn't care about you (it may not even have a notion of what is a human), you are still made of atoms it could use for something else.

taneq 7y ago
Are you still a living human if it just shoops you off into a virtual world indistinguishable from your current one, so it can use your atoms for something more productive? What about if you upgrade yourself to the point where you're no longer made of mortal meat?
- loup-vaillant 7y ago
  > Are you still a living human if it just shoops you off into a virtual world indistinguishable from your current one As far as I can assess, yes I am. (For a couple reasons, including my disbelief in philosophical zombies, I believe in mind uploading). > so it can use your atoms for something more productive? Wait a minute, running a Matrix takes energy no matter how you put it. Energy the machine could use for something else… If the machine doesn't care about me, why would it even bother? It would just take the atoms and discard any information (that is, me) that it does not need. Running a Matrix is a strong indication that the machine does, in fact, care about me. > What about if you upgrade yourself to the point where you're no longer made of mortal meat? That's more delicate. I'm not sure exactly what would ensure the continuity of my identity. Mere mind uploading most likely would, but upgrades to my own brain… I think the answer will depend on what neurosciences and such tell us.
  - taneq 7y ago
    All very good answers! For the second one I feel that a combination of scientific curiosity and altruism would be enough for most AIs up to the weakly godlike level to keep us around. For the third, I don't think continuity can be a significant factor in identity, because even we as meat-humans don't experience it, and given the possibility of physically-indistinguishable duplicates, it leans on Cartesian dualism. Also once we can upload our mind to a data file, it does weird things to a whole bunch of current morals which are based on the scarcity of any given human mind.
    - ThrowawayR2 7y ago
      > altruism Some of the worst human beings I've ever encountered were also extremely talented and intelligent engineers. There is zero reason to assume a hyperintelligent AI would automatically be altruistic.

skissane 7y ago

I think part of the problem with the question is that intelligence and motivations/drives are orthogonal. Intelligence is just a means to an end, it says nothing about the end to which it is a means. To say that a being is vastly more intelligent than human beings (super-intelligent) says nothing about what motivations it may have. Our own motivations are mostly given to us by our genetic and cultural inheritance, rather than something we've constructed for ourselves by our own intelligence. I think it likely, that any super-intelligent AI created by human beings, will be given motivations which somehow reflect the motivations and values of its creators. Not necessarily exactly the same – a silicon-based life-form has no need for food or sex as carbon-based life-forms do – but, I think a super-intelligent AI created by humans will likely be given motivations to seek to favour the interests of its creators, whether "its creators" are construed broadly (all of humanity as a whole) or narrowly (just the individual or corporation or nation-state which creates it.)