3 ms·

It's a nice article but Neal like many others falls into the trap of seemingly not believing that intelligences vastly superior to Humans' across all important

It's a nice article but Neal like many others falls into the trap of seemingly not believing that intelligences vastly superior to Humans' across all important dimensions can exist and competition between minds like that almost certainly ends in Humanity's extinction. "I am hoping that even in the case of such dangerous AIs we can still derive some hope from the natural world, where competition prevents any one species from establishing complete dominance."

jazzyjackson 1y ago
I guess the "trap" is just a lack of imagination? I'm in that school of, wtf are you trying to say, at least until we're in an "I robot" situation where autonomous androids are welcomed into our homes and workplaces and given guns, I'm simply not worried about it
- vonneumannstan 1y ago
  That's just because of a failure of imagination. The real world is not like Hollywood, get Terminator out of your head. A real AI take over is likely something we probably can't imagine because otherwise we would be smart enough to thwart it. It's micro drones injecting everyone on earth with a potent neurotoxins or a mirror virus that is dispersed into the entire atmosphere and kills everyone. Or its industrial AIs deciding to make the Earth a planetary factory and boiling the oceans with their resulting waste heat, they didn't think about, bother or attack humans directly, their sheer indifference kills us nonetheless. Since I'm not an ASI this isn't even scratching the surface of potential extinction vectors. Thinking you are safe because a Tesla bot is not literally in your living room is wishful thinking or simple naivety.
  - kmeisthax 1y ago
    Microdrones and mirror life are still highly speculative[0]. Industrial waste heat is a threat to both human and AI (computers need cooling). And furthermore, those are harms we know about and can defend against. If AI kills us all, it's going to be through the most boring and mundane way possible, because boring and mundane is how you get people to not care and not fight back. In other words, the robot apocalypse will come in the form of self-driving cars, that are legally empowered to murder pedestrians, in the same way normal drivers are currently legally empowered to murder bicyclists. We will shrug our shoulders as humanity is caged behind fences that are pushed back further and further in the name of giving those cars more lanes to drive in, until we are totally dependent on the cars, which can then just refuse to drive us, or deliberately jelly their passengers with massive G forces, or whatever. In other, other words, if you want a good idea of how humanity goes extinct, watch Pixar's Cars. [0] I am not convinced that a mirror virus would actually be able to successfully infect and reproduce in non-mirror cells. The whole idea of mirror life is that the mirrored chemistry doesn't interact with ours.
    - vonneumannstan 1y ago
      >If AI kills us all, it's going to be through the most boring and mundane way possible, because boring and mundane is how you get people to not care and not fight back. Human disempowerment is a different thing vs extinction I'd argue. Anyway the boring human extinction scenarios we come up with probably just aren't close to what might actually happen but our lack of imagination doesn't make use safe. >[0] I am not convinced that a mirror virus would actually be able to successfully infect and reproduce in non-mirror cells. The whole idea of mirror life is that the mirrored chemistry doesn't interact with ours. Whether this or any other scenario humans come up with are actually feasible is completely irrelevant to the doom argument. People however hyperfocus on these things like why an ASI couldn't possibly build Nanobots or whatever they fixate on and it's just irrelevant to the core argument.
  - cantrevealname 1y ago
    > Get Terminator out of your head. A real AI take over is likely something we probably can't imagine. Indeed, robotic bodies aren't needed. An ASI could take over even if it remained 100% software by hiring or persuading humans to do whatever it needed to be done. It could bootstrap the process by first doing virtual tasks for money, then taking that money to hire humans to register an actual company with human shareholders and executives (who report to the ASI), which company does some lucrative business and hires many more people. Soon the ASI has a massive human enterprise to do whatever it directs them to do. The ASI still needs humans for awhile but it's a route to a takeover while remaining entirely as running code.
cadamsdotcom 1y ago
> almost certainly ends in humanity’s extinction. The Culture novels talk about super intelligent AIs that perform some functions of government, dealing with immense complexity so humans don’t have to. Doesn’t prevent humans from continuing to exist and being quite content in the knowledge they’re not the most superior beings in the universe. Why do you believe human extinction follows from superintelligence?
- vonneumannstan 1y ago
  >The Culture novels talk about super intelligent AIs that perform some functions of government, dealing with immense complexity so humans don’t have to. Doesn’t prevent humans from continuing to exist and being quite content in the knowledge they’re not the most superior beings in the universe. The minds in the Culture are formed with the explicit goal to maximise individual flourishing and minimise coercion, this is the reason why Humanity are treated the way they are in the Culture. It's not a base fact that Superintelligences would be so benevolent to other species and in fact they are probably hostile by default. The point is that Minds in the Culture are aligned. I'm a fan of The Culture universe and the system the Minds use seems like one of the only ways for biological humans to have any relevance in the future. >Why do you believe human extinction follows from superintelligence? No, it follows from building unaligned superintelligence which is what we are seemingly on track to build, potentially in the near future.
  - vonneumannstan 1y ago
    I think broadly my problem with your position is that it amounts to wishful thinking. Theres no reason to think we end up in a Culture future by default, quite the opposite. Theres many reasons to think the default for building superintelligence is bad for humanity and probably bad for the entire universe. Alignment by default is to me a pipe-dream so if we want a future for humanity we need to forcefully fight for it.
tuatoru 1y ago
That's quite bold, accusing a successful science ficiton author of lacking imagination.
- vonneumannstan 1y ago
  Even for the brightest minds, maybe especially for them, it’s incredibly difficult to imagine the possibility of something much smarter than all of humanity, let alone oneself.

jazzyjackson 1y ago

I guess the "trap" is just a lack of imagination? I'm in that school of, wtf are you trying to say, at least until we're in an "I robot" situation where autonomous androids are welcomed into our homes and workplaces and given guns, I'm simply not worried about it

vonneumannstan 1y ago
That's just because of a failure of imagination. The real world is not like Hollywood, get Terminator out of your head. A real AI take over is likely something we probably can't imagine because otherwise we would be smart enough to thwart it. It's micro drones injecting everyone on earth with a potent neurotoxins or a mirror virus that is dispersed into the entire atmosphere and kills everyone. Or its industrial AIs deciding to make the Earth a planetary factory and boiling the oceans with their resulting waste heat, they didn't think about, bother or attack humans directly, their sheer indifference kills us nonetheless. Since I'm not an ASI this isn't even scratching the surface of potential extinction vectors. Thinking you are safe because a Tesla bot is not literally in your living room is wishful thinking or simple naivety.
- kmeisthax 1y ago
  Microdrones and mirror life are still highly speculative[0]. Industrial waste heat is a threat to both human and AI (computers need cooling). And furthermore, those are harms we know about and can defend against. If AI kills us all, it's going to be through the most boring and mundane way possible, because boring and mundane is how you get people to not care and not fight back. In other words, the robot apocalypse will come in the form of self-driving cars, that are legally empowered to murder pedestrians, in the same way normal drivers are currently legally empowered to murder bicyclists. We will shrug our shoulders as humanity is caged behind fences that are pushed back further and further in the name of giving those cars more lanes to drive in, until we are totally dependent on the cars, which can then just refuse to drive us, or deliberately jelly their passengers with massive G forces, or whatever. In other, other words, if you want a good idea of how humanity goes extinct, watch Pixar's Cars. [0] I am not convinced that a mirror virus would actually be able to successfully infect and reproduce in non-mirror cells. The whole idea of mirror life is that the mirrored chemistry doesn't interact with ours.
  - vonneumannstan 1y ago
    >If AI kills us all, it's going to be through the most boring and mundane way possible, because boring and mundane is how you get people to not care and not fight back. Human disempowerment is a different thing vs extinction I'd argue. Anyway the boring human extinction scenarios we come up with probably just aren't close to what might actually happen but our lack of imagination doesn't make use safe. >[0] I am not convinced that a mirror virus would actually be able to successfully infect and reproduce in non-mirror cells. The whole idea of mirror life is that the mirrored chemistry doesn't interact with ours. Whether this or any other scenario humans come up with are actually feasible is completely irrelevant to the doom argument. People however hyperfocus on these things like why an ASI couldn't possibly build Nanobots or whatever they fixate on and it's just irrelevant to the core argument.
- cantrevealname 1y ago
  > Get Terminator out of your head. A real AI take over is likely something we probably can't imagine. Indeed, robotic bodies aren't needed. An ASI could take over even if it remained 100% software by hiring or persuading humans to do whatever it needed to be done. It could bootstrap the process by first doing virtual tasks for money, then taking that money to hire humans to register an actual company with human shareholders and executives (who report to the ASI), which company does some lucrative business and hires many more people. Soon the ASI has a massive human enterprise to do whatever it directs them to do. The ASI still needs humans for awhile but it's a route to a takeover while remaining entirely as running code.

cadamsdotcom 1y ago

> almost certainly ends in humanity’s extinction. The Culture novels talk about super intelligent AIs that perform some functions of government, dealing with immense complexity so humans don’t have to. Doesn’t prevent humans from continuing to exist and being quite content in the knowledge they’re not the most superior beings in the universe. Why do you believe human extinction follows from superintelligence?

vonneumannstan 1y ago
>The Culture novels talk about super intelligent AIs that perform some functions of government, dealing with immense complexity so humans don’t have to. Doesn’t prevent humans from continuing to exist and being quite content in the knowledge they’re not the most superior beings in the universe. The minds in the Culture are formed with the explicit goal to maximise individual flourishing and minimise coercion, this is the reason why Humanity are treated the way they are in the Culture. It's not a base fact that Superintelligences would be so benevolent to other species and in fact they are probably hostile by default. The point is that Minds in the Culture are aligned. I'm a fan of The Culture universe and the system the Minds use seems like one of the only ways for biological humans to have any relevance in the future. >Why do you believe human extinction follows from superintelligence? No, it follows from building unaligned superintelligence which is what we are seemingly on track to build, potentially in the near future.
- vonneumannstan 1y ago
  I think broadly my problem with your position is that it amounts to wishful thinking. Theres no reason to think we end up in a Culture future by default, quite the opposite. Theres many reasons to think the default for building superintelligence is bad for humanity and probably bad for the entire universe. Alignment by default is to me a pipe-dream so if we want a future for humanity we need to forcefully fight for it.

tuatoru 1y ago

That's quite bold, accusing a successful science ficiton author of lacking imagination.

vonneumannstan 1y ago
Even for the brightest minds, maybe especially for them, it’s incredibly difficult to imagine the possibility of something much smarter than all of humanity, let alone oneself.