4 ms·

Here's the fundamental problem as I see it: probabilities are meaningless. We can talk about conditionals...potentials...likelihoods....whatever until we are bl

Here's the fundamental problem as I see it: probabilities are meaningless. We can talk about conditionals...potentials...likelihoods....whatever until we are blue in the face. All they do is provide a thin veneer over what I see is the truth: we simply have no real idea regarding most existential risks. Probability is a false god that justifies our wasting resources on problems we can't hope to solve, as we don't even understand them. AGI is the canonical problem of this type--absolutely no real understanding of the parameter space. What can it do...can it do this or that...what will it do...how to bound/align what it will do...???; well--just tell me what it is first. Aliens could show up tomorrow and smoke us (in theory), so what dollar amount should we spend on that (it's the same problem really...what capabilities will the aliens have so as to focus our efforts on their not smoking us). > What do you propose we spend money on that actually moves the conditional probabilities away from global nuclear war? First, let me say that I appreciate the frank discussion, and that I also appreciate the original author's willingness to author posts on this topic, these are in fact important issues. To answer your question, if it were my decision (I actually just responded to a couple RFPs with a sketch of a system for viral bioinformatics [which were declined]), I would say our largest current existential threat is our systems of government that have not made significant progress in the last couple hundred years. Regardless of how we score existential threats (I eschew probabilities personally), it's clear that we will need effective governance to respond to them as they arise (let's call covid a fire drill that we failed miserably at). We need research into questioning the fundamental assumptions by which we govern (secrecy, militancy, the paradigm of competition). How do we set goals as a world, not as nations? How do we engineer trusted environments, not zero trust environments? This is where my money would be.

benlivengood 6y ago
> AGI is the canonical problem of this type--absolutely no real understanding of the parameter space. What can it do...can it do this or that...what will it do...how to bound/align what it will do...???; well--just tell me what it is first. Aliens could show up tomorrow and smoke us (in theory), so what dollar amount should we spend on that (it's the same problem really...what capabilities will the aliens have so as to focus our efforts on their not smoking us). I'd say aliens are in the supernova and gamma ray burst territory; even if we understood a lot about them we would be mostly powerless to counter them as an existential threat because there are very few actions we could take to deflect, defeat, or escape alien attack. Reasoning with them is probably the best we could do. > I would say our largest current existential threat is our systems of government Just to be clear; do you mean completely wiping out humanity or a reset to some small population that might have another chance at it? > We need research into questioning the fundamental assumptions by which we govern (secrecy, militancy, the paradigm of competition). How do we set goals as a world, not as nations? How do we engineer trusted environments, not zero trust environments? This is were my money would be. This is a lot closer than you may realize to what a lot of AGI existential risk folks research. AGIs will almost certainly be agents in the world; able to act and smart enough to understand their ability to act and affect the world and themselves. The correspondence to governance is pretty close; humans are pretty smart actors who understand a lot of the dynamics they have control over and have goals they want to achieve. AGI will have to fit into an ecosystem of humans and their governments in a beneficial way, and a big part of solving AGI risk is understanding governance, decision theory, goals, and values well enough to not create AGI that turns into a despotic warmonger to achieve its unaligned goals. We know that human agents can end up as despots. Take the question of how to set goals as the world, not as nations. That extends down to the individual level as well; the coordination problem, unaligned incentives, the tragedy of the commons, decision theory, and other game theory problems can be extended from individuals vs. individuals to nations vs. nations. If anything, AGI will be easier to solve than human governance both because there will be fewer ethical restrictions and because we can (hopefully) engineer AGI in ways that align with human valued and incentives instead of trying to reason with humans. A trusted environment is, if I take your meaning correctly, one in which nations have aligned their incentives so closely so as to be able to trust that the actions of other nations are honest and in good faith because any other options would be counterproductive. That's at least the sense I get from trust between individuals; after enough time with someone one can be fairly certain that they have enough shared values and goals to be able to come to similar decisions as one would, and trust them to act in accordance either through mutual shared interest (including friendship, admiration, respect, etc.) or the benefit of continued cooperation in the future. I suppose my first thought is that having "nations" to begin with breaks a lot of shared incentives because it overlays unevenly distributed wealth and opportunity with arbitrary legal and social groupings that nevertheless a lot of people care very deeply about. Getting nations to truly cooperate probably means giving up a lot of identity based on tradition, origin, and history. I have no clue how best to convince people that what really matters about them and their neighbors is how individually happy, fulfilled, and free to explore they are regardless of their differences. Getting rid of false beliefs in zero sum economics and the futility of status signalling might be a start, but humans are sort of innately programmed for a lot of counterproductive beliefs about status, pride, fairness, worth, etc. Not getting too far afield, I am also curious how you reason under uncertainty without probabilities. E.g. you don't know how things may turn out but have some ideas; how do you choose what to do? How will you know if you made a good or bad choice? Is it quantifiable?
- codekilla 6y ago
  > Just to be clear; do you mean completely wiping out humanity or a reset to some small population that might have another chance at it? Either might occur, given deployment of nuclear or bio technology. > Getting nations to truly cooperate probably means giving up a lot of identity based on tradition, origin, and history. I have no clue how best to convince people that what really matters about them and their neighbors is how individually happy, fulfilled, and free to explore they are regardless of their differences. Getting rid of false beliefs in zero sum economics and the futility of status signalling might be a start, but humans are sort of innately programmed for a lot of counterproductive beliefs about status, pride, fairness, worth, etc. Yes, and I think one needs to realize that this is a 100 year project. Technologies like nuclear fusion, and quantum computing I put into this '100 year' category, that is those that would catalyze profound transformations of humanity (synthetic biology being another, Abstract General Intelligence another [I do not like the term 'artificial']) given about 100 years and sufficient resources. So rebuilding and rethinking completely the way humans govern themselves I think is one of these 100 year projects. Things like nomadic citizens (humans without borders), economics with externalities an explicit factor, zero secrecy non zero sum international relations. All stuff that is easy enough to write down, but with no real chance of swift movement towards. Future economic systems are hopefully cooperative, perhaps further enabled by greater gains in digitizing the economy (though risks towards greater inequality that is further destabilizing seem just as likely) . > reason under uncertainty without probabilities. I got started in math and computation working on quant stuff during the first financial crash. It was a front row seat to how probability goes wrong (your copulas aren't worth the pixels you graphed them on). The idea is that you have to recognize some things just aren't accurately quantifiable. If you don't recognize that, you just place yourself in a situation where you have a false sense of certainty about uncertainty. You just simply can't assign a probability accurately to anything you so choose. It's not that probability isn't worthwhile, it most certainly is, the point is to be able to accept when you are dealing with a situation where it is non-informational (and likely even counter-informational).
duvenaud 6y ago
I think everyone who uses probabilities to make decisions agrees that all our models are flawed in many ways. But how else do you propose to weigh different possibilities? As for your proposal for how to spend money, I agree those are fruitful and promising directions. I think the fields of public choice theory and mechanism design are working on some of the problems you mention.

benlivengood 6y ago

> AGI is the canonical problem of this type--absolutely no real understanding of the parameter space. What can it do...can it do this or that...what will it do...how to bound/align what it will do...???; well--just tell me what it is first. Aliens could show up tomorrow and smoke us (in theory), so what dollar amount should we spend on that (it's the same problem really...what capabilities will the aliens have so as to focus our efforts on their not smoking us). I'd say aliens are in the supernova and gamma ray burst territory; even if we understood a lot about them we would be mostly powerless to counter them as an existential threat because there are very few actions we could take to deflect, defeat, or escape alien attack. Reasoning with them is probably the best we could do. > I would say our largest current existential threat is our systems of government Just to be clear; do you mean completely wiping out humanity or a reset to some small population that might have another chance at it? > We need research into questioning the fundamental assumptions by which we govern (secrecy, militancy, the paradigm of competition). How do we set goals as a world, not as nations? How do we engineer trusted environments, not zero trust environments? This is were my money would be. This is a lot closer than you may realize to what a lot of AGI existential risk folks research. AGIs will almost certainly be agents in the world; able to act and smart enough to understand their ability to act and affect the world and themselves. The correspondence to governance is pretty close; humans are pretty smart actors who understand a lot of the dynamics they have control over and have goals they want to achieve. AGI will have to fit into an ecosystem of humans and their governments in a beneficial way, and a big part of solving AGI risk is understanding governance, decision theory, goals, and values well enough to not create AGI that turns into a despotic warmonger to achieve its unaligned goals. We know that human agents can end up as despots. Take the question of how to set goals as the world, not as nations. That extends down to the individual level as well; the coordination problem, unaligned incentives, the tragedy of the commons, decision theory, and other game theory problems can be extended from individuals vs. individuals to nations vs. nations. If anything, AGI will be easier to solve than human governance both because there will be fewer ethical restrictions and because we can (hopefully) engineer AGI in ways that align with human valued and incentives instead of trying to reason with humans. A trusted environment is, if I take your meaning correctly, one in which nations have aligned their incentives so closely so as to be able to trust that the actions of other nations are honest and in good faith because any other options would be counterproductive. That's at least the sense I get from trust between individuals; after enough time with someone one can be fairly certain that they have enough shared values and goals to be able to come to similar decisions as one would, and trust them to act in accordance either through mutual shared interest (including friendship, admiration, respect, etc.) or the benefit of continued cooperation in the future. I suppose my first thought is that having "nations" to begin with breaks a lot of shared incentives because it overlays unevenly distributed wealth and opportunity with arbitrary legal and social groupings that nevertheless a lot of people care very deeply about. Getting nations to truly cooperate probably means giving up a lot of identity based on tradition, origin, and history. I have no clue how best to convince people that what really matters about them and their neighbors is how individually happy, fulfilled, and free to explore they are regardless of their differences. Getting rid of false beliefs in zero sum economics and the futility of status signalling might be a start, but humans are sort of innately programmed for a lot of counterproductive beliefs about status, pride, fairness, worth, etc. Not getting too far afield, I am also curious how you reason under uncertainty without probabilities. E.g. you don't know how things may turn out but have some ideas; how do you choose what to do? How will you know if you made a good or bad choice? Is it quantifiable?

codekilla 6y ago
> Just to be clear; do you mean completely wiping out humanity or a reset to some small population that might have another chance at it? Either might occur, given deployment of nuclear or bio technology. > Getting nations to truly cooperate probably means giving up a lot of identity based on tradition, origin, and history. I have no clue how best to convince people that what really matters about them and their neighbors is how individually happy, fulfilled, and free to explore they are regardless of their differences. Getting rid of false beliefs in zero sum economics and the futility of status signalling might be a start, but humans are sort of innately programmed for a lot of counterproductive beliefs about status, pride, fairness, worth, etc. Yes, and I think one needs to realize that this is a 100 year project. Technologies like nuclear fusion, and quantum computing I put into this '100 year' category, that is those that would catalyze profound transformations of humanity (synthetic biology being another, Abstract General Intelligence another [I do not like the term 'artificial']) given about 100 years and sufficient resources. So rebuilding and rethinking completely the way humans govern themselves I think is one of these 100 year projects. Things like nomadic citizens (humans without borders), economics with externalities an explicit factor, zero secrecy non zero sum international relations. All stuff that is easy enough to write down, but with no real chance of swift movement towards. Future economic systems are hopefully cooperative, perhaps further enabled by greater gains in digitizing the economy (though risks towards greater inequality that is further destabilizing seem just as likely) . > reason under uncertainty without probabilities. I got started in math and computation working on quant stuff during the first financial crash. It was a front row seat to how probability goes wrong (your copulas aren't worth the pixels you graphed them on). The idea is that you have to recognize some things just aren't accurately quantifiable. If you don't recognize that, you just place yourself in a situation where you have a false sense of certainty about uncertainty. You just simply can't assign a probability accurately to anything you so choose. It's not that probability isn't worthwhile, it most certainly is, the point is to be able to accept when you are dealing with a situation where it is non-informational (and likely even counter-informational).

duvenaud 6y ago

I think everyone who uses probabilities to make decisions agrees that all our models are flawed in many ways. But how else do you propose to weigh different possibilities? As for your proposal for how to spend money, I agree those are fruitful and promising directions. I think the fields of public choice theory and mechanism design are working on some of the problems you mention.