5 ms·

Probably not a bad heuristic: unless proven, don't assume existential risk.

by RandomLensman 3y ago

altpaddle 3y ago
Dude just think about that for a moment. By definition if existential risk has been proven. It's already too late
- RandomLensman 3y ago
  Totally not true: take nuclear weapons, for example, or a large meteorite impact.
  - ludwik 3y ago
    So what do you mean when you say that the "risk is proven"? If by "the risk is proven" you mean there's more than a 0% chance of an event happening, then there are almost an infinite number of such risks. There is certainly more than a 0% risk of humanity facing severe problems with an unaligned AGI in the future. If it means the event happening is certain (100%), then neither a meteorite impact (of a magnitude harmful to humanity) nor the actual use of nuclear weapons fall into this category. If you're referring only to risks of events that have occurred at least once in the past (as inferred from your examples), then we would be unprepared for any new risks. In my opinion, it's much more complicated. There is no clear-cut category of "proven risks" that allows us to disregard other dangers and justifiably see those concerned about them as crazy radicals. We must assess each potential risk individually, estimating both the probability of the event (which in almost all cases will be neither 100% nor 0%) and the potential harm it could cause. Different people naturally come up with different estimates, leading to various priorities in preventing different kinds of risks.
    - RandomLensman 3y ago
      No, I mean that there is a proven way for the risk to materialise, not just some tall tale. Tall tales might(!) justify some caution, but they are a very different class of issue. Biological risks are perhaps in the latter category. Also, as we don't know the probabilities, I don't think they are a useful metric. Made up numbers don't help there. Edit: I would encourage people to study some classic cold war thinking, because that relied little on probabilities, but rather on trying to avoid situations where stability is lost, leading to nuclear war (a known existential risk).
      - ludwik 3y ago
        "there is a proven way for the risk to materialise" - I still don't know what this means. "Proven" how? Wouldn't your edit apply to any not-impossible risk (i.e., > 0% probability)? For example, "trying to avoid situations where control over AGI is lost, leading to unaligned AGI (a known existential risk)"? You can not run away from having to estimate how likely the risk is to happen (in addition to being "known").
        RandomLensman 3y ago
        Proven means all parts needed for the realisation of the risk are known and shown to exist (at least in principle, in a lab etc.). There can be some middle ground where a large part is known and shown to exist (biological risks, for example).), but not all. No in relation to my edit, because we have no existing mechanism for the AGI risk to happen. We have hypotheses about what an AGI could or could not do. It could all be incorrect. Playing around with likelihoods that have no basis in reality isn't helping there. Where we have known and fully understood risks and we can actually estimate a probability there we might use that somewhat to guide efforts (but that invites potentially complacency that is deadly).
  - richardw 3y ago
    Nukes and meteorites have very few components that are hard to predict. One goes bang almost entirely on command and the other follows Newton's laws of motion. Neither actively tries to effect any change in the world, so the risk is only "can we spot a meteorite early enough". Once we do, it doesn't try to evade us or take another shot at goal. A better example might be covid, which was very mildly more unpredictable than a meteor, and changed its code very slowly in a purely random fashion, and we had many historical examples of how to combat.
_Algernon_ 3y ago
Existential risks are usually proven by the subject being extinct at which point no action can be taken to prevent it. Reasoning about tiny probabilities of massive (or infinite) cost is hard because the expected value is large, but just gambling on it not happening is almost certain to work out. We should still make attempts at incorporating them into decision making because tiny yearly probabilities are still virtually certain to occur at larger time scales (eg. 100s-1000s of years).
- RandomLensman 3y ago
  Are we extinct? No. Could a large impact kill us all? Yes. Expected value and probability have no place in these discussions. Some risks we know can materialize, for others we have perhaps a story on what could happen. We need to clearly distinguish between where there is a proven mechanism for doom vs where there is not.
  - _Algernon_ 3y ago
    >We need to clearly distinguish between where there is a proven mechanism for doom vs where there is not. How do you prove a mechanism for doom without it already having occurred? The existential risk is completely orthogonal to whether it has already happened, and generally action can only be taken to prevent or mitigate before it happens. Having the foresight to mitigate future problems is a good thing and should be encouraged. >Expected value and probability have no place in these discussions. I disagree. Expected value and probability is a framework for decision making in uncertain environments. They certainly have a place in these discussions.
    - RandomLensman 3y ago
      I disagree that there is orthogonality. Have we killed us all with nuclear weapons, for example? Anyone can make up any story - at the very least there needs to be a proven mechanism. The precautionary principle is not useful when facing totally hypothetically issues. People purposefully avoided probabilities in high risk existential situations in the past. There is only one path of events and we need to manage that one.

Probably not a bad heuristic: unless proven, don't assume existential risk.

altpaddle 3y ago

Dude just think about that for a moment. By definition if existential risk has been proven. It's already too late

RandomLensman 3y ago
Totally not true: take nuclear weapons, for example, or a large meteorite impact.
- ludwik 3y ago
  So what do you mean when you say that the "risk is proven"? If by "the risk is proven" you mean there's more than a 0% chance of an event happening, then there are almost an infinite number of such risks. There is certainly more than a 0% risk of humanity facing severe problems with an unaligned AGI in the future. If it means the event happening is certain (100%), then neither a meteorite impact (of a magnitude harmful to humanity) nor the actual use of nuclear weapons fall into this category. If you're referring only to risks of events that have occurred at least once in the past (as inferred from your examples), then we would be unprepared for any new risks. In my opinion, it's much more complicated. There is no clear-cut category of "proven risks" that allows us to disregard other dangers and justifiably see those concerned about them as crazy radicals. We must assess each potential risk individually, estimating both the probability of the event (which in almost all cases will be neither 100% nor 0%) and the potential harm it could cause. Different people naturally come up with different estimates, leading to various priorities in preventing different kinds of risks.
  - RandomLensman 3y ago
    No, I mean that there is a proven way for the risk to materialise, not just some tall tale. Tall tales might(!) justify some caution, but they are a very different class of issue. Biological risks are perhaps in the latter category. Also, as we don't know the probabilities, I don't think they are a useful metric. Made up numbers don't help there. Edit: I would encourage people to study some classic cold war thinking, because that relied little on probabilities, but rather on trying to avoid situations where stability is lost, leading to nuclear war (a known existential risk).
    - ludwik 3y ago
      "there is a proven way for the risk to materialise" - I still don't know what this means. "Proven" how? Wouldn't your edit apply to any not-impossible risk (i.e., > 0% probability)? For example, "trying to avoid situations where control over AGI is lost, leading to unaligned AGI (a known existential risk)"? You can not run away from having to estimate how likely the risk is to happen (in addition to being "known").
      - RandomLensman 3y ago
        Proven means all parts needed for the realisation of the risk are known and shown to exist (at least in principle, in a lab etc.). There can be some middle ground where a large part is known and shown to exist (biological risks, for example).), but not all. No in relation to my edit, because we have no existing mechanism for the AGI risk to happen. We have hypotheses about what an AGI could or could not do. It could all be incorrect. Playing around with likelihoods that have no basis in reality isn't helping there. Where we have known and fully understood risks and we can actually estimate a probability there we might use that somewhat to guide efforts (but that invites potentially complacency that is deadly).
- richardw 3y ago
  Nukes and meteorites have very few components that are hard to predict. One goes bang almost entirely on command and the other follows Newton's laws of motion. Neither actively tries to effect any change in the world, so the risk is only "can we spot a meteorite early enough". Once we do, it doesn't try to evade us or take another shot at goal. A better example might be covid, which was very mildly more unpredictable than a meteor, and changed its code very slowly in a purely random fashion, and we had many historical examples of how to combat.

_Algernon_ 3y ago

Existential risks are usually proven by the subject being extinct at which point no action can be taken to prevent it. Reasoning about tiny probabilities of massive (or infinite) cost is hard because the expected value is large, but just gambling on it not happening is almost certain to work out. We should still make attempts at incorporating them into decision making because tiny yearly probabilities are still virtually certain to occur at larger time scales (eg. 100s-1000s of years).

RandomLensman 3y ago
Are we extinct? No. Could a large impact kill us all? Yes. Expected value and probability have no place in these discussions. Some risks we know can materialize, for others we have perhaps a story on what could happen. We need to clearly distinguish between where there is a proven mechanism for doom vs where there is not.
- _Algernon_ 3y ago
  >We need to clearly distinguish between where there is a proven mechanism for doom vs where there is not. How do you prove a mechanism for doom without it already having occurred? The existential risk is completely orthogonal to whether it has already happened, and generally action can only be taken to prevent or mitigate before it happens. Having the foresight to mitigate future problems is a good thing and should be encouraged. >Expected value and probability have no place in these discussions. I disagree. Expected value and probability is a framework for decision making in uncertain environments. They certainly have a place in these discussions.
  - RandomLensman 3y ago
    I disagree that there is orthogonality. Have we killed us all with nuclear weapons, for example? Anyone can make up any story - at the very least there needs to be a proven mechanism. The precautionary principle is not useful when facing totally hypothetically issues. People purposefully avoided probabilities in high risk existential situations in the past. There is only one path of events and we need to manage that one.