3 ms·

> extremely bad and worth avoiding Shrugs???? What exactly do you mean? Quantify this, how bad...with respect to AGI? > discourages anyone from preparing for

> extremely bad and worth avoiding Shrugs???? What exactly do you mean? Quantify this, how bad...with respect to AGI? > discourages anyone from preparing for potential nuclear wars Interesting, what would be your suggestions for preparing? Mine would be: AT ALL COST AVOID. (I mean it...ALL COST). Anything else, and I would classify it as a species wide failure to adapt to technology (In this case EXISTENTIALLY dangerous tech). Further, there is no way to grasp or accurately predict confounding factors of post total nuclear war that would put humankind at further risk.

benlivengood 6y ago
> Interesting, what would be your suggestions for preparing? Mine would be: AT ALL COST AVOID. (I mean it...ALL COST). This gets at the heart of existential risk. Should we abandon all technology to avoid the possibility of nuclear war and just wait for an asteroid or climate change or resource exhaustion to kill us? Nuclear destruction only gets easier over time as technology progresses. My cell phone is faster than the supercomputers of the 80's used to model nuclear explosions. Even North Korea can refine uranium. Guided rockets are cheap enough to launch high school science projects. Hypersonic cruise missiles are on the horizon. Generally I would propose we avoid nuclear war being an existential risk by going to the stars as soon as possible. That has other costs that, yes, should be balanced against the costs of nuclear war. AGI is another path to a post-existential-threat future for humanity but probably has the greatest risks because it accelerates virtually all of the existing risky technology and climate trends, balanced only by the hope that its intelligence grows far faster than the risks and that it's value-aligned with humanity.
- codekilla 6y ago
  > AGI is another path to a post-existential-threat future for humanity but probably has the greatest risks because it accelerates virtually all of the existing risky technology and climate trends No, it is not. The reason is this: it is science fiction. This makes it no less, or no more, dangerous than anything else I can make up with a quasi realistic chance of existing in the near term future. When thermonuclear bombs were detonated, they thought there was a possibility that the reaction would get away from them, and incinerate the atmosphere. The first test, they were scared this actually happened as the explosion carried on longer than they expected. The point? They actually assigned a probability (non zero) to this event, and then carried on with the test anyway. So I could say that a hydrogen bomb, using some iteration of technology just beyond our grasp is likely to kill every human on earth in the near future. Am I right? Who knows..but in conjecturing this I am no more or 'less wrong' than anyone saying AGI is the greatest threat.
  - benlivengood 6y ago
    If we're comparing instantaneous risks, then I agree with your arguments. The most likely thing that could kill the most people today (in the next 24 hours) is a large nuclear war, followed by things like a nearby supernova, gamma ray burst, etc. An asteroid large enough to wipe out half of humanity before Saturday is really unlikely to be close enough to not have been observed already. In the first atomic bomb test the atmospheric ignition risk was not a conditional probability over the next 50 years or more but an estimate of P(ignition | nuclear detonation next week). Very tightly bound immediate risk. P(detonation next week) was ~1. When risk is estimated over the next 50 or 100 years the probabilities change quite a bit, especially because it's possible (though unlikely) that every nation cooperatively disarms its nuclear stockpile. Disarmament is not possible today so it can't count against that risk. It is a factor in the 50-year risk. Climate change, AGI, asteroids, pandemics, biowarfare, etc. all become potential risks in the longer time scales, but their conditional probabilities are much fuzzier. That doesn't mean it's useless to try to measure the probability; when assigning dollars to counter risk it makes sense to spend relative to the expected utility loss of each potential outcome, times the probability of being able to change the outcome. There's very little reason to spend money on short timescales to avoid a nearby supernova or a gamma ray burst pointed at us. There's nothing we can do. We can do things about nuclear weapons, climate change, AGI, etc. But we need accurate probabilies to avoid the failure of devoting 100% of GDP to the current biggest risk. AGI is not a threat within the next day or probably the next year, and even if it was it would be almost certain that events are already in motion and nothing could be done. Dealing with AGI risk means spending an appropriate amount of money to offset risks that might appear more than a year or two out. Searching the skies for unknown asteroids is worthwhile because only with enough advance warning can anything be done, and it's quite likely we could avoid a strike with enough warning (years). It's in everyone's best interest to disarm every nuclear weapon but there's a strong defection incentive to be the last one with them which we also have to be realistic about. Achieving zero nuclear weapons is likely to require a one-world government which is unachievable in any short or medium timeframe. What do you propose we spend money on that actually moves the conditional probabilities away from global nuclear war? Do those proposals (even with ludicrous amounts of money) have a higher likelihood of success than risk-dollars-adjusted reduction of climate change for example? If not, then we can't prevent nuclear war at any cost as you originally proposed. In fact we can't prevent nuclear war in the first place; there is no set of concrete steps (short of globally synchronized sabotage) to take this year or even next. Human beings will have their fingers on the nuclear triggers for the foreseeable future. We have to mitigate the other risks that we can right now otherwise it's a waste of money. Hopefully as the world becomes more stable it will eventually become possible to actually prevent nuclear war through disarmament.
- Retric 6y ago
  You’re making the common mistake to assume AGI significantly increases the advancement of technology. If AI research has demonstrated anything it’s Intelligence doesn’t increase linearly with computation. Compare two identical chess engines where one has 10% more computational power behind it and that program has a surprisingly limited increase in ELO score. It’s seductive to think AGI will suddenly advance technology dramatically, but dumping more effort into existing technology has a bad habit of hitting diminishing returns. Ideas like nano-machines seem to have vast potential, until you realize biology is already operating at those scales and has significant limitations.
  - benlivengood 6y ago
    > It’s seductive to think AGI will suddenly advance technology dramatically, but dumping more effort into existing technology has a bad habit of hitting diminishing returns. Ideas like nano-machines seem to have vast potential, until you realize biology is already operating at those scales and has significant limitations. There are a few reasons to think that AGI will significantly outproduce humans; it's likely cheap to run compared to the training costs. GPT-3 can generate text quite a bit faster than a human at maybe a billionth of the cost of its training, for example. Once achieved it's likely that practical progress on a lot of problems will actually be pretty rapid. Even if we only achieve parity with human intelligence and can never improve AGI beyond that we'll experience a few orders of magnitude in speedup of research. I think it's a little early to write off nanotechnology when we haven't seen much more than tech demos at this point. Biology is a soup of proteins using brownian motion to find reaction sites. Highly impressive and amazing to be sure but needs to be wet, fed, and in a narrow temperature band. Precision molecular engineering is a game-changer in so many ways.
    - Retric 6y ago
      Producing text more cheaply is a poor metric for technological advancement. Going to the basics of say energy production, heat engines are already close to theoretical limitations. We already have 60+% efficient power plant so nothing can double future efficiency, and even in theory hitting 100% is never going to happen. The world has better technology and more infrastructure than 10 years ago, but it’s almost entirely evolutionary not revolutionary. Even the tantalizing self driving cars in the near future come with a massive downside of congestion from cars driving without people in them. As to nanotechnology, we already design molecules to interface with biology via synthetic drugs, so essentially biological nanotechnology is going to be about creating larger structures not smaller ones.

benlivengood 6y ago

> Interesting, what would be your suggestions for preparing? Mine would be: AT ALL COST AVOID. (I mean it...ALL COST). This gets at the heart of existential risk. Should we abandon all technology to avoid the possibility of nuclear war and just wait for an asteroid or climate change or resource exhaustion to kill us? Nuclear destruction only gets easier over time as technology progresses. My cell phone is faster than the supercomputers of the 80's used to model nuclear explosions. Even North Korea can refine uranium. Guided rockets are cheap enough to launch high school science projects. Hypersonic cruise missiles are on the horizon. Generally I would propose we avoid nuclear war being an existential risk by going to the stars as soon as possible. That has other costs that, yes, should be balanced against the costs of nuclear war. AGI is another path to a post-existential-threat future for humanity but probably has the greatest risks because it accelerates virtually all of the existing risky technology and climate trends, balanced only by the hope that its intelligence grows far faster than the risks and that it's value-aligned with humanity.

codekilla 6y ago
> AGI is another path to a post-existential-threat future for humanity but probably has the greatest risks because it accelerates virtually all of the existing risky technology and climate trends No, it is not. The reason is this: it is science fiction. This makes it no less, or no more, dangerous than anything else I can make up with a quasi realistic chance of existing in the near term future. When thermonuclear bombs were detonated, they thought there was a possibility that the reaction would get away from them, and incinerate the atmosphere. The first test, they were scared this actually happened as the explosion carried on longer than they expected. The point? They actually assigned a probability (non zero) to this event, and then carried on with the test anyway. So I could say that a hydrogen bomb, using some iteration of technology just beyond our grasp is likely to kill every human on earth in the near future. Am I right? Who knows..but in conjecturing this I am no more or 'less wrong' than anyone saying AGI is the greatest threat.
- benlivengood 6y ago
  If we're comparing instantaneous risks, then I agree with your arguments. The most likely thing that could kill the most people today (in the next 24 hours) is a large nuclear war, followed by things like a nearby supernova, gamma ray burst, etc. An asteroid large enough to wipe out half of humanity before Saturday is really unlikely to be close enough to not have been observed already. In the first atomic bomb test the atmospheric ignition risk was not a conditional probability over the next 50 years or more but an estimate of P(ignition | nuclear detonation next week). Very tightly bound immediate risk. P(detonation next week) was ~1. When risk is estimated over the next 50 or 100 years the probabilities change quite a bit, especially because it's possible (though unlikely) that every nation cooperatively disarms its nuclear stockpile. Disarmament is not possible today so it can't count against that risk. It is a factor in the 50-year risk. Climate change, AGI, asteroids, pandemics, biowarfare, etc. all become potential risks in the longer time scales, but their conditional probabilities are much fuzzier. That doesn't mean it's useless to try to measure the probability; when assigning dollars to counter risk it makes sense to spend relative to the expected utility loss of each potential outcome, times the probability of being able to change the outcome. There's very little reason to spend money on short timescales to avoid a nearby supernova or a gamma ray burst pointed at us. There's nothing we can do. We can do things about nuclear weapons, climate change, AGI, etc. But we need accurate probabilies to avoid the failure of devoting 100% of GDP to the current biggest risk. AGI is not a threat within the next day or probably the next year, and even if it was it would be almost certain that events are already in motion and nothing could be done. Dealing with AGI risk means spending an appropriate amount of money to offset risks that might appear more than a year or two out. Searching the skies for unknown asteroids is worthwhile because only with enough advance warning can anything be done, and it's quite likely we could avoid a strike with enough warning (years). It's in everyone's best interest to disarm every nuclear weapon but there's a strong defection incentive to be the last one with them which we also have to be realistic about. Achieving zero nuclear weapons is likely to require a one-world government which is unachievable in any short or medium timeframe. What do you propose we spend money on that actually moves the conditional probabilities away from global nuclear war? Do those proposals (even with ludicrous amounts of money) have a higher likelihood of success than risk-dollars-adjusted reduction of climate change for example? If not, then we can't prevent nuclear war at any cost as you originally proposed. In fact we can't prevent nuclear war in the first place; there is no set of concrete steps (short of globally synchronized sabotage) to take this year or even next. Human beings will have their fingers on the nuclear triggers for the foreseeable future. We have to mitigate the other risks that we can right now otherwise it's a waste of money. Hopefully as the world becomes more stable it will eventually become possible to actually prevent nuclear war through disarmament.
Retric 6y ago
You’re making the common mistake to assume AGI significantly increases the advancement of technology. If AI research has demonstrated anything it’s Intelligence doesn’t increase linearly with computation. Compare two identical chess engines where one has 10% more computational power behind it and that program has a surprisingly limited increase in ELO score. It’s seductive to think AGI will suddenly advance technology dramatically, but dumping more effort into existing technology has a bad habit of hitting diminishing returns. Ideas like nano-machines seem to have vast potential, until you realize biology is already operating at those scales and has significant limitations.
- benlivengood 6y ago
  > It’s seductive to think AGI will suddenly advance technology dramatically, but dumping more effort into existing technology has a bad habit of hitting diminishing returns. Ideas like nano-machines seem to have vast potential, until you realize biology is already operating at those scales and has significant limitations. There are a few reasons to think that AGI will significantly outproduce humans; it's likely cheap to run compared to the training costs. GPT-3 can generate text quite a bit faster than a human at maybe a billionth of the cost of its training, for example. Once achieved it's likely that practical progress on a lot of problems will actually be pretty rapid. Even if we only achieve parity with human intelligence and can never improve AGI beyond that we'll experience a few orders of magnitude in speedup of research. I think it's a little early to write off nanotechnology when we haven't seen much more than tech demos at this point. Biology is a soup of proteins using brownian motion to find reaction sites. Highly impressive and amazing to be sure but needs to be wet, fed, and in a narrow temperature band. Precision molecular engineering is a game-changer in so many ways.
  - Retric 6y ago
    Producing text more cheaply is a poor metric for technological advancement. Going to the basics of say energy production, heat engines are already close to theoretical limitations. We already have 60+% efficient power plant so nothing can double future efficiency, and even in theory hitting 100% is never going to happen. The world has better technology and more infrastructure than 10 years ago, but it’s almost entirely evolutionary not revolutionary. Even the tantalizing self driving cars in the near future come with a massive downside of congestion from cars driving without people in them. As to nanotechnology, we already design molecules to interface with biology via synthetic drugs, so essentially biological nanotechnology is going to be about creating larger structures not smaller ones.