2 ms·

> what separates it from religion? The rational methods used to establish a large P(doom). To definitively falsify the claim, humanity would need to live alon

by rphv 16d ago

> what separates it from religion? The rational methods used to establish a large P(doom). To definitively falsify the claim, humanity would need to live alongside superintelligence forever without it wiping us out. That doesn't mean we can't be pretty darn sure though, in the same way that we can be pretty darn sure that nuclear war could end civilization.

nullc 15d ago
Those "rational methods" are essentially just pascal's wager. You stick some ~infinitely bad outcome (nasal daemons torturing simulations of you and everyone you love for all time) and/or ~infinitely good outcome (perfect paradise for as many sentient minds as could fit in the physical universe), then you can get whatever conclusion you want by multiplying any finite probability by these things, and even give yourself bonus asspats for being conservative by adding several order of magnitude fudge factors against your claims. The rationalist doom and sex cult uses convoluted mathematics inspired language to justify their faith-- but at the end of the day it's just faith no matter how they dress it up.
- hollerith 15d ago
  No, it's not Pascal's wager because Pascal's wager works even when the probabilities are tiny whereas Eliezer's p(doom) is about 99% unless AI "progress" is shut down and kept shut long enough that the current crop of AI researchers is going to have to find something else to do for a career. In other words, it's not Pascal's wager if the very bad outcome is what you naturally expect to happen. And what I just wrote applies to not just Eliezer and his allies: I don't know a single person or a single published argument that has ever used Pascal's-wager-like argumentation to argue that AI "progress" needs to be stopped. Eliezer is the author of a blog post titled "Pascal's mugging", but that blog post and any follow-up discussion are completely silent on the dangers of AI.
hollerith 15d ago
I wish you'd used "very large asteroid strike on Earth" rather than "nuclear war" because although nuclear war will probably make thing "uncivilized" for a while, I'm pretty sure human civilization bounces right back after even the largest realistic nuclear war.
- dTal 15d ago
  I wouldn't be so sure about that. 95% of the population will starve to death, global supply chains will collapse, and the world will regress to a pre-industrial-revolution lifestyle - with the critical difference that all shallow oil and coal deposits have been mined out, not to mention many other resources. You're going to have a very tough time bootstrapping it all again. All vital knowledge for running a modern civilization will be lost - most of it isn't even written down now, but is instead implicit in distributed social structures which will all be destroyed. They won't re-form quickly either - travel will be even more risky than in the Dark Ages thanks to fallout. Even a healthy society can coast in a mostly steady state for thousands of years without technological takeoff, and we've had several civilizations in the past who seemed to hold most of the pieces - empiricism, healthy economy transcending food production, precision metalworking - and yet never seemed to quite get there. This won't even be that - it's a restart in a world depleted of natural resources and blighted by death, radiation, mass extinction, and climate change. I don't think we get a second shot.
  - hollerith 15d ago
    "95% of the population will starve to death" is very unlikely although I can't completely rule it out as a possibility. The research on nuclear winter is of a poor quality, and we should be funding better research. "The world will regress to a pre-industrial-revolution lifestyle" seems to be a retreat from "civilization will end" (not that I agree with it). >travel will be even more risky than in the Dark Ages thanks to fallout The radioactive contamination caused by Chernobyl has persisted for decades. In contrast, the radionuclides created by nuclear explosions are of a very different nature and persist for weeks. 3 weeks after the end of the explosions, there will be no place except ground-zero craters (i.e., very small areas) where the levels of radioactivity remain at deadly levels. (But I definitely agree with you that during those critical 3 weeks, almost all people who decide to travel will end up dying unless they know a lot about nuclear war and have good radiation detectors.) To repeat my overall point: the worst-case scenario for the outcome of AI research should be compared to an asteroid strike by an extremely large asteroid, not to nuclear war—because nuclear war is too tame in its effects to make a good comparison. The reason I got involved in this discussion is that some people use the possibility of nuclear war as one reason for us to go forward with AI research as fast as possible. Such people argue that a sufficiently mature AI technology would eliminate the possibility of nuclear war. I think that is a fallacious argument become AI "progress" is the more potent of the two dangers.

by rphv 16d ago

nullc 15d ago

Those "rational methods" are essentially just pascal's wager. You stick some ~infinitely bad outcome (nasal daemons torturing simulations of you and everyone you love for all time) and/or ~infinitely good outcome (perfect paradise for as many sentient minds as could fit in the physical universe), then you can get whatever conclusion you want by multiplying any finite probability by these things, and even give yourself bonus asspats for being conservative by adding several order of magnitude fudge factors against your claims. The rationalist doom and sex cult uses convoluted mathematics inspired language to justify their faith-- but at the end of the day it's just faith no matter how they dress it up.

hollerith 15d ago
No, it's not Pascal's wager because Pascal's wager works even when the probabilities are tiny whereas Eliezer's p(doom) is about 99% unless AI "progress" is shut down and kept shut long enough that the current crop of AI researchers is going to have to find something else to do for a career. In other words, it's not Pascal's wager if the very bad outcome is what you naturally expect to happen. And what I just wrote applies to not just Eliezer and his allies: I don't know a single person or a single published argument that has ever used Pascal's-wager-like argumentation to argue that AI "progress" needs to be stopped. Eliezer is the author of a blog post titled "Pascal's mugging", but that blog post and any follow-up discussion are completely silent on the dangers of AI.

hollerith 15d ago

I wish you'd used "very large asteroid strike on Earth" rather than "nuclear war" because although nuclear war will probably make thing "uncivilized" for a while, I'm pretty sure human civilization bounces right back after even the largest realistic nuclear war.

dTal 15d ago
I wouldn't be so sure about that. 95% of the population will starve to death, global supply chains will collapse, and the world will regress to a pre-industrial-revolution lifestyle - with the critical difference that all shallow oil and coal deposits have been mined out, not to mention many other resources. You're going to have a very tough time bootstrapping it all again. All vital knowledge for running a modern civilization will be lost - most of it isn't even written down now, but is instead implicit in distributed social structures which will all be destroyed. They won't re-form quickly either - travel will be even more risky than in the Dark Ages thanks to fallout. Even a healthy society can coast in a mostly steady state for thousands of years without technological takeoff, and we've had several civilizations in the past who seemed to hold most of the pieces - empiricism, healthy economy transcending food production, precision metalworking - and yet never seemed to quite get there. This won't even be that - it's a restart in a world depleted of natural resources and blighted by death, radiation, mass extinction, and climate change. I don't think we get a second shot.
- hollerith 15d ago
  "95% of the population will starve to death" is very unlikely although I can't completely rule it out as a possibility. The research on nuclear winter is of a poor quality, and we should be funding better research. "The world will regress to a pre-industrial-revolution lifestyle" seems to be a retreat from "civilization will end" (not that I agree with it). >travel will be even more risky than in the Dark Ages thanks to fallout The radioactive contamination caused by Chernobyl has persisted for decades. In contrast, the radionuclides created by nuclear explosions are of a very different nature and persist for weeks. 3 weeks after the end of the explosions, there will be no place except ground-zero craters (i.e., very small areas) where the levels of radioactivity remain at deadly levels. (But I definitely agree with you that during those critical 3 weeks, almost all people who decide to travel will end up dying unless they know a lot about nuclear war and have good radiation detectors.) To repeat my overall point: the worst-case scenario for the outcome of AI research should be compared to an asteroid strike by an extremely large asteroid, not to nuclear war—because nuclear war is too tame in its effects to make a good comparison. The reason I got involved in this discussion is that some people use the possibility of nuclear war as one reason for us to go forward with AI research as fast as possible. Such people argue that a sufficiently mature AI technology would eliminate the possibility of nuclear war. I think that is a fallacious argument become AI "progress" is the more potent of the two dangers.