3 ms·

> AGI is another path to a post-existential-threat future for humanity but probably has the greatest risks because it accelerates virtually all of the existing

> AGI is another path to a post-existential-threat future for humanity but probably has the greatest risks because it accelerates virtually all of the existing risky technology and climate trends No, it is not. The reason is this: it is science fiction. This makes it no less, or no more, dangerous than anything else I can make up with a quasi realistic chance of existing in the near term future. When thermonuclear bombs were detonated, they thought there was a possibility that the reaction would get away from them, and incinerate the atmosphere. The first test, they were scared this actually happened as the explosion carried on longer than they expected. The point? They actually assigned a probability (non zero) to this event, and then carried on with the test anyway. So I could say that a hydrogen bomb, using some iteration of technology just beyond our grasp is likely to kill every human on earth in the near future. Am I right? Who knows..but in conjecturing this I am no more or 'less wrong' than anyone saying AGI is the greatest threat.

benlivengood 6y ago
If we're comparing instantaneous risks, then I agree with your arguments. The most likely thing that could kill the most people today (in the next 24 hours) is a large nuclear war, followed by things like a nearby supernova, gamma ray burst, etc. An asteroid large enough to wipe out half of humanity before Saturday is really unlikely to be close enough to not have been observed already. In the first atomic bomb test the atmospheric ignition risk was not a conditional probability over the next 50 years or more but an estimate of P(ignition | nuclear detonation next week). Very tightly bound immediate risk. P(detonation next week) was ~1. When risk is estimated over the next 50 or 100 years the probabilities change quite a bit, especially because it's possible (though unlikely) that every nation cooperatively disarms its nuclear stockpile. Disarmament is not possible today so it can't count against that risk. It is a factor in the 50-year risk. Climate change, AGI, asteroids, pandemics, biowarfare, etc. all become potential risks in the longer time scales, but their conditional probabilities are much fuzzier. That doesn't mean it's useless to try to measure the probability; when assigning dollars to counter risk it makes sense to spend relative to the expected utility loss of each potential outcome, times the probability of being able to change the outcome. There's very little reason to spend money on short timescales to avoid a nearby supernova or a gamma ray burst pointed at us. There's nothing we can do. We can do things about nuclear weapons, climate change, AGI, etc. But we need accurate probabilies to avoid the failure of devoting 100% of GDP to the current biggest risk. AGI is not a threat within the next day or probably the next year, and even if it was it would be almost certain that events are already in motion and nothing could be done. Dealing with AGI risk means spending an appropriate amount of money to offset risks that might appear more than a year or two out. Searching the skies for unknown asteroids is worthwhile because only with enough advance warning can anything be done, and it's quite likely we could avoid a strike with enough warning (years). It's in everyone's best interest to disarm every nuclear weapon but there's a strong defection incentive to be the last one with them which we also have to be realistic about. Achieving zero nuclear weapons is likely to require a one-world government which is unachievable in any short or medium timeframe. What do you propose we spend money on that actually moves the conditional probabilities away from global nuclear war? Do those proposals (even with ludicrous amounts of money) have a higher likelihood of success than risk-dollars-adjusted reduction of climate change for example? If not, then we can't prevent nuclear war at any cost as you originally proposed. In fact we can't prevent nuclear war in the first place; there is no set of concrete steps (short of globally synchronized sabotage) to take this year or even next. Human beings will have their fingers on the nuclear triggers for the foreseeable future. We have to mitigate the other risks that we can right now otherwise it's a waste of money. Hopefully as the world becomes more stable it will eventually become possible to actually prevent nuclear war through disarmament.
- codekilla 6y ago
  Here's the fundamental problem as I see it: probabilities are meaningless. We can talk about conditionals...potentials...likelihoods....whatever until we are blue in the face. All they do is provide a thin veneer over what I see is the truth: we simply have no real idea regarding most existential risks. Probability is a false god that justifies our wasting resources on problems we can't hope to solve, as we don't even understand them. AGI is the canonical problem of this type--absolutely no real understanding of the parameter space. What can it do...can it do this or that...what will it do...how to bound/align what it will do...???; well--just tell me what it is first. Aliens could show up tomorrow and smoke us (in theory), so what dollar amount should we spend on that (it's the same problem really...what capabilities will the aliens have so as to focus our efforts on their not smoking us). > What do you propose we spend money on that actually moves the conditional probabilities away from global nuclear war? First, let me say that I appreciate the frank discussion, and that I also appreciate the original author's willingness to author posts on this topic, these are in fact important issues. To answer your question, if it were my decision (I actually just responded to a couple RFPs with a sketch of a system for viral bioinformatics [which were declined]), I would say our largest current existential threat is our systems of government that have not made significant progress in the last couple hundred years. Regardless of how we score existential threats (I eschew probabilities personally), it's clear that we will need effective governance to respond to them as they arise (let's call covid a fire drill that we failed miserably at). We need research into questioning the fundamental assumptions by which we govern (secrecy, militancy, the paradigm of competition). How do we set goals as a world, not as nations? How do we engineer trusted environments, not zero trust environments? This is where my money would be.
  - benlivengood 6y ago
    > AGI is the canonical problem of this type--absolutely no real understanding of the parameter space. What can it do...can it do this or that...what will it do...how to bound/align what it will do...???; well--just tell me what it is first. Aliens could show up tomorrow and smoke us (in theory), so what dollar amount should we spend on that (it's the same problem really...what capabilities will the aliens have so as to focus our efforts on their not smoking us). I'd say aliens are in the supernova and gamma ray burst territory; even if we understood a lot about them we would be mostly powerless to counter them as an existential threat because there are very few actions we could take to deflect, defeat, or escape alien attack. Reasoning with them is probably the best we could do. > I would say our largest current existential threat is our systems of government Just to be clear; do you mean completely wiping out humanity or a reset to some small population that might have another chance at it? > We need research into questioning the fundamental assumptions by which we govern (secrecy, militancy, the paradigm of competition). How do we set goals as a world, not as nations? How do we engineer trusted environments, not zero trust environments? This is were my money would be. This is a lot closer than you may realize to what a lot of AGI existential risk folks research. AGIs will almost certainly be agents in the world; able to act and smart enough to understand their ability to act and affect the world and themselves. The correspondence to governance is pretty close; humans are pretty smart actors who understand a lot of the dynamics they have control over and have goals they want to achieve. AGI will have to fit into an ecosystem of humans and their governments in a beneficial way, and a big part of solving AGI risk is understanding governance, decision theory, goals, and values well enough to not create AGI that turns into a despotic warmonger to achieve its unaligned goals. We know that human agents can end up as despots. Take the question of how to set goals as the world, not as nations. That extends down to the individual level as well; the coordination problem, unaligned incentives, the tragedy of the commons, decision theory, and other game theory problems can be extended from individuals vs. individuals to nations vs. nations. If anything, AGI will be easier to solve than human governance both because there will be fewer ethical restrictions and because we can (hopefully) engineer AGI in ways that align with human valued and incentives instead of trying to reason with humans. A trusted environment is, if I take your meaning correctly, one in which nations have aligned their incentives so closely so as to be able to trust that the actions of other nations are honest and in good faith because any other options would be counterproductive. That's at least the sense I get from trust between individuals; after enough time with someone one can be fairly certain that they have enough shared values and goals to be able to come to similar decisions as one would, and trust them to act in accordance either through mutual shared interest (including friendship, admiration, respect, etc.) or the benefit of continued cooperation in the future. I suppose my first thought is that having "nations" to begin with breaks a lot of shared incentives because it overlays unevenly distributed wealth and opportunity with arbitrary legal and social groupings that nevertheless a lot of people care very deeply about. Getting nations to truly cooperate probably means giving up a lot of identity based on tradition, origin, and history. I have no clue how best to convince people that what really matters about them and their neighbors is how individually happy, fulfilled, and free to explore they are regardless of their differences. Getting rid of false beliefs in zero sum economics and the futility of status signalling might be a start, but humans are sort of innately programmed for a lot of counterproductive beliefs about status, pride, fairness, worth, etc. Not getting too far afield, I am also curious how you reason under uncertainty without probabilities. E.g. you don't know how things may turn out but have some ideas; how do you choose what to do? How will you know if you made a good or bad choice? Is it quantifiable?
XorNot 6y ago
Urgh, atmospheric ignition was considered but it was studied 6 months before the first nuclear test was conducted: [1] https://www.realclearscience.com/blog/2019/09/12/the_fear_that_a_nuclear_bomb_could_ignite_the_atmosphere.html https://www.realclearscience.com/blog/2019/09/12/the_fear_th... The whole atmospheric ignition thing depends on physics which just doesn't work: specifically, that you can reach a temperature without gravitational pressure that triggers nuclear fusion at ambient atmospheric pressures. It is in the same category as the "concerns" the LHC would create black holes which swallowed the Earth.
- codekilla 6y ago
  That’s not how it was presented in [0], which is one book I had read, among others. And that’s fine if that particular argument is incorrect, the logic of the point I’m making still stands. I can envision a feasible chance of a technological improvement to explosives that would make them ‘completely’ dangerous (kill every last human) if that’s what we’re going for. [0] https://anniejacobsen.com/pentagons-brain/ https://anniejacobsen.com/pentagons-brain/

benlivengood 6y ago

If we're comparing instantaneous risks, then I agree with your arguments. The most likely thing that could kill the most people today (in the next 24 hours) is a large nuclear war, followed by things like a nearby supernova, gamma ray burst, etc. An asteroid large enough to wipe out half of humanity before Saturday is really unlikely to be close enough to not have been observed already. In the first atomic bomb test the atmospheric ignition risk was not a conditional probability over the next 50 years or more but an estimate of P(ignition | nuclear detonation next week). Very tightly bound immediate risk. P(detonation next week) was ~1. When risk is estimated over the next 50 or 100 years the probabilities change quite a bit, especially because it's possible (though unlikely) that every nation cooperatively disarms its nuclear stockpile. Disarmament is not possible today so it can't count against that risk. It is a factor in the 50-year risk. Climate change, AGI, asteroids, pandemics, biowarfare, etc. all become potential risks in the longer time scales, but their conditional probabilities are much fuzzier. That doesn't mean it's useless to try to measure the probability; when assigning dollars to counter risk it makes sense to spend relative to the expected utility loss of each potential outcome, times the probability of being able to change the outcome. There's very little reason to spend money on short timescales to avoid a nearby supernova or a gamma ray burst pointed at us. There's nothing we can do. We can do things about nuclear weapons, climate change, AGI, etc. But we need accurate probabilies to avoid the failure of devoting 100% of GDP to the current biggest risk. AGI is not a threat within the next day or probably the next year, and even if it was it would be almost certain that events are already in motion and nothing could be done. Dealing with AGI risk means spending an appropriate amount of money to offset risks that might appear more than a year or two out. Searching the skies for unknown asteroids is worthwhile because only with enough advance warning can anything be done, and it's quite likely we could avoid a strike with enough warning (years). It's in everyone's best interest to disarm every nuclear weapon but there's a strong defection incentive to be the last one with them which we also have to be realistic about. Achieving zero nuclear weapons is likely to require a one-world government which is unachievable in any short or medium timeframe. What do you propose we spend money on that actually moves the conditional probabilities away from global nuclear war? Do those proposals (even with ludicrous amounts of money) have a higher likelihood of success than risk-dollars-adjusted reduction of climate change for example? If not, then we can't prevent nuclear war at any cost as you originally proposed. In fact we can't prevent nuclear war in the first place; there is no set of concrete steps (short of globally synchronized sabotage) to take this year or even next. Human beings will have their fingers on the nuclear triggers for the foreseeable future. We have to mitigate the other risks that we can right now otherwise it's a waste of money. Hopefully as the world becomes more stable it will eventually become possible to actually prevent nuclear war through disarmament.

codekilla 6y ago
Here's the fundamental problem as I see it: probabilities are meaningless. We can talk about conditionals...potentials...likelihoods....whatever until we are blue in the face. All they do is provide a thin veneer over what I see is the truth: we simply have no real idea regarding most existential risks. Probability is a false god that justifies our wasting resources on problems we can't hope to solve, as we don't even understand them. AGI is the canonical problem of this type--absolutely no real understanding of the parameter space. What can it do...can it do this or that...what will it do...how to bound/align what it will do...???; well--just tell me what it is first. Aliens could show up tomorrow and smoke us (in theory), so what dollar amount should we spend on that (it's the same problem really...what capabilities will the aliens have so as to focus our efforts on their not smoking us). > What do you propose we spend money on that actually moves the conditional probabilities away from global nuclear war? First, let me say that I appreciate the frank discussion, and that I also appreciate the original author's willingness to author posts on this topic, these are in fact important issues. To answer your question, if it were my decision (I actually just responded to a couple RFPs with a sketch of a system for viral bioinformatics [which were declined]), I would say our largest current existential threat is our systems of government that have not made significant progress in the last couple hundred years. Regardless of how we score existential threats (I eschew probabilities personally), it's clear that we will need effective governance to respond to them as they arise (let's call covid a fire drill that we failed miserably at). We need research into questioning the fundamental assumptions by which we govern (secrecy, militancy, the paradigm of competition). How do we set goals as a world, not as nations? How do we engineer trusted environments, not zero trust environments? This is where my money would be.
- benlivengood 6y ago
  > AGI is the canonical problem of this type--absolutely no real understanding of the parameter space. What can it do...can it do this or that...what will it do...how to bound/align what it will do...???; well--just tell me what it is first. Aliens could show up tomorrow and smoke us (in theory), so what dollar amount should we spend on that (it's the same problem really...what capabilities will the aliens have so as to focus our efforts on their not smoking us). I'd say aliens are in the supernova and gamma ray burst territory; even if we understood a lot about them we would be mostly powerless to counter them as an existential threat because there are very few actions we could take to deflect, defeat, or escape alien attack. Reasoning with them is probably the best we could do. > I would say our largest current existential threat is our systems of government Just to be clear; do you mean completely wiping out humanity or a reset to some small population that might have another chance at it? > We need research into questioning the fundamental assumptions by which we govern (secrecy, militancy, the paradigm of competition). How do we set goals as a world, not as nations? How do we engineer trusted environments, not zero trust environments? This is were my money would be. This is a lot closer than you may realize to what a lot of AGI existential risk folks research. AGIs will almost certainly be agents in the world; able to act and smart enough to understand their ability to act and affect the world and themselves. The correspondence to governance is pretty close; humans are pretty smart actors who understand a lot of the dynamics they have control over and have goals they want to achieve. AGI will have to fit into an ecosystem of humans and their governments in a beneficial way, and a big part of solving AGI risk is understanding governance, decision theory, goals, and values well enough to not create AGI that turns into a despotic warmonger to achieve its unaligned goals. We know that human agents can end up as despots. Take the question of how to set goals as the world, not as nations. That extends down to the individual level as well; the coordination problem, unaligned incentives, the tragedy of the commons, decision theory, and other game theory problems can be extended from individuals vs. individuals to nations vs. nations. If anything, AGI will be easier to solve than human governance both because there will be fewer ethical restrictions and because we can (hopefully) engineer AGI in ways that align with human valued and incentives instead of trying to reason with humans. A trusted environment is, if I take your meaning correctly, one in which nations have aligned their incentives so closely so as to be able to trust that the actions of other nations are honest and in good faith because any other options would be counterproductive. That's at least the sense I get from trust between individuals; after enough time with someone one can be fairly certain that they have enough shared values and goals to be able to come to similar decisions as one would, and trust them to act in accordance either through mutual shared interest (including friendship, admiration, respect, etc.) or the benefit of continued cooperation in the future. I suppose my first thought is that having "nations" to begin with breaks a lot of shared incentives because it overlays unevenly distributed wealth and opportunity with arbitrary legal and social groupings that nevertheless a lot of people care very deeply about. Getting nations to truly cooperate probably means giving up a lot of identity based on tradition, origin, and history. I have no clue how best to convince people that what really matters about them and their neighbors is how individually happy, fulfilled, and free to explore they are regardless of their differences. Getting rid of false beliefs in zero sum economics and the futility of status signalling might be a start, but humans are sort of innately programmed for a lot of counterproductive beliefs about status, pride, fairness, worth, etc. Not getting too far afield, I am also curious how you reason under uncertainty without probabilities. E.g. you don't know how things may turn out but have some ideas; how do you choose what to do? How will you know if you made a good or bad choice? Is it quantifiable?

XorNot 6y ago

Urgh, atmospheric ignition was considered but it was studied 6 months before the first nuclear test was conducted: [1] https://www.realclearscience.com/blog/2019/09/12/the_fear_that_a_nuclear_bomb_could_ignite_the_atmosphere.html https://www.realclearscience.com/blog/2019/09/12/the_fear_th... The whole atmospheric ignition thing depends on physics which just doesn't work: specifically, that you can reach a temperature without gravitational pressure that triggers nuclear fusion at ambient atmospheric pressures. It is in the same category as the "concerns" the LHC would create black holes which swallowed the Earth.

codekilla 6y ago
That’s not how it was presented in [0], which is one book I had read, among others. And that’s fine if that particular argument is incorrect, the logic of the point I’m making still stands. I can envision a feasible chance of a technological improvement to explosives that would make them ‘completely’ dangerous (kill every last human) if that’s what we’re going for. [0] https://anniejacobsen.com/pentagons-brain/ https://anniejacobsen.com/pentagons-brain/