5 ms·
> If you truly believed that Superhuman AI was near, and it could act with malice, won't you try to slow things down a bit? FWIW, that's called zealotry and pe
by swatcoder 3y ago
> If you truly believed that Superhuman AI was near, and it could act with malice, won't you try to slow things down a bit?
FWIW, that's called zealotry and people do a lot of dramatic, disruptive things in the name of it. It may be rightly aimed and save the world (or whatever you care about), but it's more often a signal to really reflect on whether you, individually, have really found yourself at the make-or-break nexus of human existence. The answer seems to be "no" most of the time.
- jacobedawson 3y agoIt's more often a signal to really reflect on whether you, individually as a Thanksgiving turkey, have really found yourself at the make-or-break nexus of turkey existence. The answer seems to be "no" most of the time.
- mlyle 3y agoYour comment perfectly justifies never worrying at all about the potential for existential or major risks; after all, one would be wrong most of the time and just engaging in zealotry.
- RandomLensman 3y agoProbably not a bad heuristic: unless proven, don't assume existential risk.
- altpaddle 3y agoDude just think about that for a moment. By definition if existential risk has been proven. It's already too late
- RandomLensman 3y agoTotally not true: take nuclear weapons, for example, or a large meteorite impact.
- ludwik 3y agoSo what do you mean when you say that the "risk is proven"? If by "the risk is proven" you mean there's more than a 0% chance of an event happening, then there are almost an infinite number of such risks. There is certainly more than a 0% risk of humanity facing severe problems with an unaligned AGI in the future. If it means the event happening is certain (100%), then neither a meteorite impact (of a magnitude harmful to humanity) nor the actual use of nuclear weapons fall into this category. If you're referring only to risks of events that have occurred at least once in the past (as inferred from your examples), then we would be unprepared for any new risks. In my opinion, it's much more complicated. There is no clear-cut category of "proven risks" that allows us to disregard other dangers and justifiably see those concerned about them as crazy radicals. We must assess each potential risk individually, estimating both the probability of the event (which in almost all cases will be neither 100% nor 0%) and the potential harm it could cause. Different people naturally come up with different estimates, leading to various priorities in preventing different kinds of risks.
- RandomLensman 3y agoNo, I mean that there is a proven way for the risk to materialise, not just some tall tale. Tall tales might(!) justify some caution, but they are a very different class of issue. Biological risks are perhaps in the latter category. Also, as we don't know the probabilities, I don't think they are a useful metric. Made up numbers don't help there. Edit: I would encourage people to study some classic cold war thinking, because that relied little on probabilities, but rather on trying to avoid situations where stability is lost, leading to nuclear war (a known existential risk).
- ludwik 3y ago"there is a proven way for the risk to materialise" - I still don't know what this means. "Proven" how? Wouldn't your edit apply to any not-impossible risk (i.e., > 0% probability)? For example, "trying to avoid situations where control over AGI is lost, leading to unaligned AGI (a known existential risk)"? You can not run away from having to estimate how likely the risk is to happen (in addition to being "known").
- _Algernon_ 3y agoExistential risks are usually proven by the subject being extinct at which point no action can be taken to prevent it. Reasoning about tiny probabilities of massive (or infinite) cost is hard because the expected value is large, but just gambling on it not happening is almost certain to work out. We should still make attempts at incorporating them into decision making because tiny yearly probabilities are still virtually certain to occur at larger time scales (eg. 100s-1000s of years).
- RandomLensman 3y agoAre we extinct? No. Could a large impact kill us all? Yes. Expected value and probability have no place in these discussions. Some risks we know can materialize, for others we have perhaps a story on what could happen. We need to clearly distinguish between where there is a proven mechanism for doom vs where there is not.
- _Algernon_ 3y ago>We need to clearly distinguish between where there is a proven mechanism for doom vs where there is not. How do you prove a mechanism for doom without it already having occurred? The existential risk is completely orthogonal to whether it has already happened, and generally action can only be taken to prevent or mitigate before it happens. Having the foresight to mitigate future problems is a good thing and should be encouraged. >Expected value and probability have no place in these discussions. I disagree. Expected value and probability is a framework for decision making in uncertain environments. They certainly have a place in these discussions.
- RandomLensman 3y agoI disagree that there is orthogonality. Have we killed us all with nuclear weapons, for example? Anyone can make up any story - at the very least there needs to be a proven mechanism. The precautionary principle is not useful when facing totally hypothetically issues. People purposefully avoided probabilities in high risk existential situations in the past. There is only one path of events and we need to manage that one.
- lewhoo 3y agoFWIW, that's called zealotry and people do a lot of dramatic, disruptive things in the name of it. That would be a really bad take on climate change.