5 ms·

The Dangerous Ideas of “Longtermism” and “Existential Risk”

23 points by shafyy 3y ago

dekhn 3y ago
Just remember, some time in the past there was a "crazy" person saying we should prepare for bad seasons by storing food and securing water, and a bunch of other people who killed the "crazy" person for having dangerous ideas.
- a_shovel 3y ago
  I'm no historian, but I'm pretty sure that storing food for hard times is something humans have done at least since the invention of pottery.
  - retrac 3y ago
    I believe they're referring to that cranky guy who argued to save even more. Every culture has one, I'm sure. I'll be that guy today. Every industrialized country should have decades worth of dried food stockpiled for every person. We have barely enough food stored to survive one major crop failure! That's insanely close to the edge considering our capacity. When I suggest this, people mostly dismiss it as bizarre and just too outside-the-box to really appreciate what I'm even suggesting. They don't live in the same world I do with the same perceived risks. To the people I've suggested it to, a volcanic eruption disrupting agriculture for years just isn't real, not in a way that would make them want to save food in planning for it. Just as a 1 in 1000 year harsh winter or drought wasn't real to a lot of neolithic farmers. Until it was. And I'm sure one of their neighbours probably even suggested the possibility to them.
    - jstarfish 3y ago
      > Every culture has one, I'm sure. I'll be that guy today. Every industrialized country should have decades worth of dried food stockpiled for every person. The Mormons come closest-- they stockpile for a year. They're a good resource when you need a shopping list for the next pandemic: https://www.churchofjesuschrist.org/study/ensign/2006/03/random-sampler/food-storage-for-one-year?lang=eng https://www.churchofjesuschrist.org/study/ensign/2006/03/ran... Decades is too long. Nothing lasts forever, even if it's the container itself degrading. You get rat infestation, rot, botulism, etc.
  - dekhn 3y ago
    Let me rewrite this to be a bit more explicit: For every cranky guy warning against "apparently theoretical problem which is not immediately affecfting our daily life", society will contain some other people who criticize the idea for being wasteful, or anticipating problems that won't occur, or won't have nearly the impact claimed, and will push back. For some subset of the warnings, they are legitimate and represent rational response to rare, but impactful events. Identifying and planning for existential threats is something that successful societies do. Only extremely unlucky societies that fail to plan survive while those that do plan are statistically more likely to survive. I don't think we exist in a time and place where if you say "we should multi-home humanity to solve the meteor problem" will lead to people spending trillions on mars bases, though.
Dig1t 3y ago
This article had the opposite of its intended effect on me. Longtermism seems like pretty sound thinking. I don't know why the article calls it a religion, it just seems like a good idea.
- PaulHoule 3y ago
  With all their trigger warnings for things that deviate slightly from the classic Yudkowsky line, Less Wrong seems to be a center for free thinking right up there with a weekend that half the conference center is rented out to transsexual maximalists and the other half is rented out to anti-vaxxers. But seriously, betting it all on a very rare outcome that requires multiple unknowns to line up doesn't make a lot of sense. I could just as easily make up 10 other improbable scenarios of effective measure zero that demand people's attention as much.
  - handoflixue 3y ago
    Keep in mind, every scenario that Less Wrong warns about (nuclear war, climate change, bio-terrorisms, AI risk, etc.) is backed up by a large number of experts in the field. Can you point to even one improbable scenario you made up, that has anywhere near that level of expert agreement?
    - PaulHoule 3y ago
      The underlying crazy scenario is that they say you have to evaluate all of the above relative to the possibility of losing a glorious future where we colonize the galaxy with self-replicating machines and put Dyson spheres around all of the stars, and can then simulate millions of copies of all the people who had the foresight to be cryonically frozen or something… And practically those folks are dismissive of climate change relative to the others and are generally not interested in technology relevant to their “long term” vision such as space industrialization, rather they spam us will trolley problems in an attempt to erase our moral sensibilities. Overall there is a religious orientation to the problem and a complete indifference to the complex social and technical problems of, for instance, switching to a carbon neutral energy system and otherwise managing risks.
krunck 3y ago
The article drips with bias. I gave up reading and will research this myself.
FranchuFranchu 3y ago
I would take longtermists seriously if they were one of those poor people they talk so much about. But right now, there is an obvious conflict of interest when they speak about longtermism.
- handoflixue 3y ago
  Keep in mind that Longtermism is a branch of Effective Altruism, which is also doing things like supporting Malaria nets to save lives: https://forum.effectivealtruism.org/topics/against-malaria-foundation https://forum.effectivealtruism.org/topics/against-malaria-f....
  - xg15 3y ago
    The Anthroposophy movement has also produced a number of unquestionably good things - i.e. Waldorf schools - but still has an esoteric quasi-religion at its core. One does not preclude the other.
- Dig1t 3y ago
  I think we should be able to talk about ideas independently of the people who are talking about them. A speaker's biases have no bearing on the actual merits of the idea.
  - FranchuFranchu 3y ago
    I agree with you regarding many areas of thought, where conclusions can be reached by deductive or inductive reasoning from premises. However, these people are proposing a "new axiom set" of sorts, where lives now have the same utility as the possibility of saving life in the future. Ethical principles have no further premises and are derived entirely from one's ideas and feelings. The ideas and feelings here are obviously biased. I think we should look critically at ideas that spring up from biased people.
NumberWangMan 3y ago
To say that climate change is not an "existential risk" is not the same as saying that it's not a massive problem, or that we should not address it, right?
- handoflixue 3y ago
  Correct. An existential risk is something that threatens to wipe out all of humanity - "extinction risk" would probably be a better term.
  - Lutger 3y ago
    According to the article, in longtermism 'existential risk' is something more specific: everything that threatens our long term 'potential'. So if becoming 'a multi planetary species' as Musk puts it is an essential part of our potential, destroying in whatever way our capability of achieving that is putting us at an existential risk. Not because we might all die on this planet, but just for the very reasons that we stay stuck here at the limits of earth.
    - handoflixue 3y ago
      Existential Risk includes the edge-case that we might damage civilization so badly that we're trapped on this planet until the sun goes supernova, yes. Or become nothing more than pets to some AI. But it's a pretty niche edge case - if we meaningfully continue as a species, expanding out into space seems pretty inevitable. And, honestly, I don't think anyone really views the edge case of "we collapsed back to medieval times" as a good ending. In particular, there's nothing about our "long term potential" that requires us to get off Earth today, or even in the next century or two - that's why it's called long-term potential.
    - helen___keller 3y ago
      This is the same thing as extinction. If humans don’t escape earth, then with probability 1 we will go extinct relatively soon (relative to the lifetime of the universe). If not from a standard extinction event like meteors and volcanoes, then from the sun going supernova (ETA 5 billion years)
Lutger 3y ago
So longtermism seems to be basically a crude form of utilitarianism that takes all possible future human-like life forms as input to its happiness calculation? This means it multiples all problems of utilitarianism by the uncertainty of predicting the future. This sound like an insane ethical framework, though I'm sure the actual writings of longtermists are more nuanced that what is portrayed in the article, I hope.
- helen___keller 3y ago
  It’s kind of “genius” as a hack to solve utilitarianisms problems - we can label all potential future utility as effectively infinite and thus any extinction scenario as a utility of negative infinite. Then, anything that influences odds of extinction a nonzero amount (say, invention of AI) multiplied by the negative infinity utility to have itself negative infinite utility. It doesn’t actually matter what the odds are, because the factor of all future utility is so powerful. So utilitarianism’s primary problem - we can’t make a consistent, calculable, useful, and agreeable utility function to aid decision making - is solved by overruling everything with the risk of extinction (consistent, calculable insofar as making a plausible argument that odds of extinction from X are greater than zero, useful that it informs how to proceed wrt X, and arguably agreeable because everyone knows extinction bad) Anyways my personal POV is that a good utility function should discount future utility, which brings us back to the incalculability issue. But if you treat future utility undiscounted, it makes it very easy to convince yourself of crazy things
  - moritzwarhier 3y ago
    Thanks for explaining why this is an apocalyptic religion. It's all there: - reliance on apocalyptic fantasies to defend unethical decisions (while actual suffering and real threats flourish) - inability to comprehend pragmatic humanistic and ethical thinking unless it's justified by irrational utopian fantasies (e.g. space civilization) It's some kind of a perverse attempt to define agnostic morals. But sincerely, I don't even believe that - it's more like a modern scam religion to distract from actual problems. E.g, AI could be used to limit resource consumption, provide efficient public transport, feed the world. Instead it'll be used to maximize resource consumption of rich people (cough, autonomous self-driving electric cars, but not for everyone...)
agalunar 3y ago
Something I've never quite understood is the (tacit) judgment that the end of the human race would be a bad thing, and that creating more people is inherently a good thing. And in fact, that it would maybe be acceptable to allow a large fraction of people to suffer if it ensured the propagation of humanity. It makes sense why this is such a common judgment – individuals with the drive to reproduce and ensure the perpetuation of their progeny are more likely to have surviving progeny, so the population is dominated by this kind of individual. But people talk about a species as if it was a living individual. It's not! a species can't feel hurt; its members can feel hurt. A species cannot prefer to continue to live (over the millenia); its members can prefer to live (for the duration of their own time on earth). In some cases talking about a species is just metonymy, or shorthand, but in other cases I don't think that usage fits. For example, if "we must ensure the human race persists" is just shorthand, it's shorthand for "people should have children", and it's strange to insist that other persons have the personal desire to have children. When someone is suffering, e.g. from depression, we can encourage them to continue to live for hope of a time where they aren't suffering (by helping them imagine a plausible, happier future), which to them might be worth enduring their present circumstances. In other words, we want to help them make a better-informed decision about their own life or death by reminding them that their current outlook is strongly colored by their current mental state. But when it comes to a species, the future will not be experienced by any of its currently living members – it will be experienced by individuals who don't yet exist! And if they are never born, they've not been robbed of life or had a preference violated (because there's no one to rob or disregard). The philosophy is much more complicated than I'm presenting, but I wanted to say something on the matter, if only because it's the sort of questioning I wish I saw more often.
- ianburrell 3y ago
  One thing that annoys me about longtermism is that they focus on the survival of human race. I can't tell if this means humans as currently exist or humans as will exist in the future. Because humans will change a lot conquering the galaxy and surviving for a long span of time. How much change is allowed before considered non-human? Do all human descendants count? Does this include modifications or cyborgs? Or uploaded minds? Most importantly, they don't seem to think of AIs as human descendants. Is it better for human-like AIs to conquer the galaxy, and/or for humans to be trapped on Earth for a million years? If that is the only choice.

23 points by shafyy 3y ago

Just remember, some time in the past there was a "crazy" person saying we should prepare for bad seasons by storing food and securing water, and a bunch of other people who killed the "crazy" person for having dangerous ideas.

a_shovel 3y ago
I'm no historian, but I'm pretty sure that storing food for hard times is something humans have done at least since the invention of pottery.
- retrac 3y ago
  I believe they're referring to that cranky guy who argued to save even more. Every culture has one, I'm sure. I'll be that guy today. Every industrialized country should have decades worth of dried food stockpiled for every person. We have barely enough food stored to survive one major crop failure! That's insanely close to the edge considering our capacity. When I suggest this, people mostly dismiss it as bizarre and just too outside-the-box to really appreciate what I'm even suggesting. They don't live in the same world I do with the same perceived risks. To the people I've suggested it to, a volcanic eruption disrupting agriculture for years just isn't real, not in a way that would make them want to save food in planning for it. Just as a 1 in 1000 year harsh winter or drought wasn't real to a lot of neolithic farmers. Until it was. And I'm sure one of their neighbours probably even suggested the possibility to them.
  - jstarfish 3y ago
    > Every culture has one, I'm sure. I'll be that guy today. Every industrialized country should have decades worth of dried food stockpiled for every person. The Mormons come closest-- they stockpile for a year. They're a good resource when you need a shopping list for the next pandemic: https://www.churchofjesuschrist.org/study/ensign/2006/03/random-sampler/food-storage-for-one-year?lang=eng https://www.churchofjesuschrist.org/study/ensign/2006/03/ran... Decades is too long. Nothing lasts forever, even if it's the container itself degrading. You get rat infestation, rot, botulism, etc.
- dekhn 3y ago
  Let me rewrite this to be a bit more explicit: For every cranky guy warning against "apparently theoretical problem which is not immediately affecfting our daily life", society will contain some other people who criticize the idea for being wasteful, or anticipating problems that won't occur, or won't have nearly the impact claimed, and will push back. For some subset of the warnings, they are legitimate and represent rational response to rare, but impactful events. Identifying and planning for existential threats is something that successful societies do. Only extremely unlucky societies that fail to plan survive while those that do plan are statistically more likely to survive. I don't think we exist in a time and place where if you say "we should multi-home humanity to solve the meteor problem" will lead to people spending trillions on mars bases, though.

Dig1t 3y ago

This article had the opposite of its intended effect on me. Longtermism seems like pretty sound thinking. I don't know why the article calls it a religion, it just seems like a good idea.

PaulHoule 3y ago
With all their trigger warnings for things that deviate slightly from the classic Yudkowsky line, Less Wrong seems to be a center for free thinking right up there with a weekend that half the conference center is rented out to transsexual maximalists and the other half is rented out to anti-vaxxers. But seriously, betting it all on a very rare outcome that requires multiple unknowns to line up doesn't make a lot of sense. I could just as easily make up 10 other improbable scenarios of effective measure zero that demand people's attention as much.
- handoflixue 3y ago
  Keep in mind, every scenario that Less Wrong warns about (nuclear war, climate change, bio-terrorisms, AI risk, etc.) is backed up by a large number of experts in the field. Can you point to even one improbable scenario you made up, that has anywhere near that level of expert agreement?
  - PaulHoule 3y ago
    The underlying crazy scenario is that they say you have to evaluate all of the above relative to the possibility of losing a glorious future where we colonize the galaxy with self-replicating machines and put Dyson spheres around all of the stars, and can then simulate millions of copies of all the people who had the foresight to be cryonically frozen or something… And practically those folks are dismissive of climate change relative to the others and are generally not interested in technology relevant to their “long term” vision such as space industrialization, rather they spam us will trolley problems in an attempt to erase our moral sensibilities. Overall there is a religious orientation to the problem and a complete indifference to the complex social and technical problems of, for instance, switching to a carbon neutral energy system and otherwise managing risks.

krunck 3y ago

The article drips with bias. I gave up reading and will research this myself.

FranchuFranchu 3y ago

I would take longtermists seriously if they were one of those poor people they talk so much about. But right now, there is an obvious conflict of interest when they speak about longtermism.

handoflixue 3y ago
Keep in mind that Longtermism is a branch of Effective Altruism, which is also doing things like supporting Malaria nets to save lives: https://forum.effectivealtruism.org/topics/against-malaria-foundation https://forum.effectivealtruism.org/topics/against-malaria-f....
- xg15 3y ago
  The Anthroposophy movement has also produced a number of unquestionably good things - i.e. Waldorf schools - but still has an esoteric quasi-religion at its core. One does not preclude the other.
Dig1t 3y ago
I think we should be able to talk about ideas independently of the people who are talking about them. A speaker's biases have no bearing on the actual merits of the idea.
- FranchuFranchu 3y ago
  I agree with you regarding many areas of thought, where conclusions can be reached by deductive or inductive reasoning from premises. However, these people are proposing a "new axiom set" of sorts, where lives now have the same utility as the possibility of saving life in the future. Ethical principles have no further premises and are derived entirely from one's ideas and feelings. The ideas and feelings here are obviously biased. I think we should look critically at ideas that spring up from biased people.

NumberWangMan 3y ago

To say that climate change is not an "existential risk" is not the same as saying that it's not a massive problem, or that we should not address it, right?

handoflixue 3y ago
Correct. An existential risk is something that threatens to wipe out all of humanity - "extinction risk" would probably be a better term.
- Lutger 3y ago
  According to the article, in longtermism 'existential risk' is something more specific: everything that threatens our long term 'potential'. So if becoming 'a multi planetary species' as Musk puts it is an essential part of our potential, destroying in whatever way our capability of achieving that is putting us at an existential risk. Not because we might all die on this planet, but just for the very reasons that we stay stuck here at the limits of earth.
  - handoflixue 3y ago
    Existential Risk includes the edge-case that we might damage civilization so badly that we're trapped on this planet until the sun goes supernova, yes. Or become nothing more than pets to some AI. But it's a pretty niche edge case - if we meaningfully continue as a species, expanding out into space seems pretty inevitable. And, honestly, I don't think anyone really views the edge case of "we collapsed back to medieval times" as a good ending. In particular, there's nothing about our "long term potential" that requires us to get off Earth today, or even in the next century or two - that's why it's called long-term potential.
  - helen___keller 3y ago
    This is the same thing as extinction. If humans don’t escape earth, then with probability 1 we will go extinct relatively soon (relative to the lifetime of the universe). If not from a standard extinction event like meteors and volcanoes, then from the sun going supernova (ETA 5 billion years)

Lutger 3y ago

So longtermism seems to be basically a crude form of utilitarianism that takes all possible future human-like life forms as input to its happiness calculation? This means it multiples all problems of utilitarianism by the uncertainty of predicting the future. This sound like an insane ethical framework, though I'm sure the actual writings of longtermists are more nuanced that what is portrayed in the article, I hope.

helen___keller 3y ago
It’s kind of “genius” as a hack to solve utilitarianisms problems - we can label all potential future utility as effectively infinite and thus any extinction scenario as a utility of negative infinite. Then, anything that influences odds of extinction a nonzero amount (say, invention of AI) multiplied by the negative infinity utility to have itself negative infinite utility. It doesn’t actually matter what the odds are, because the factor of all future utility is so powerful. So utilitarianism’s primary problem - we can’t make a consistent, calculable, useful, and agreeable utility function to aid decision making - is solved by overruling everything with the risk of extinction (consistent, calculable insofar as making a plausible argument that odds of extinction from X are greater than zero, useful that it informs how to proceed wrt X, and arguably agreeable because everyone knows extinction bad) Anyways my personal POV is that a good utility function should discount future utility, which brings us back to the incalculability issue. But if you treat future utility undiscounted, it makes it very easy to convince yourself of crazy things
- moritzwarhier 3y ago
  Thanks for explaining why this is an apocalyptic religion. It's all there: - reliance on apocalyptic fantasies to defend unethical decisions (while actual suffering and real threats flourish) - inability to comprehend pragmatic humanistic and ethical thinking unless it's justified by irrational utopian fantasies (e.g. space civilization) It's some kind of a perverse attempt to define agnostic morals. But sincerely, I don't even believe that - it's more like a modern scam religion to distract from actual problems. E.g, AI could be used to limit resource consumption, provide efficient public transport, feed the world. Instead it'll be used to maximize resource consumption of rich people (cough, autonomous self-driving electric cars, but not for everyone...)

agalunar 3y ago

Something I've never quite understood is the (tacit) judgment that the end of the human race would be a bad thing, and that creating more people is inherently a good thing. And in fact, that it would maybe be acceptable to allow a large fraction of people to suffer if it ensured the propagation of humanity. It makes sense why this is such a common judgment – individuals with the drive to reproduce and ensure the perpetuation of their progeny are more likely to have surviving progeny, so the population is dominated by this kind of individual. But people talk about a species as if it was a living individual. It's not! a species can't feel hurt; its members can feel hurt. A species cannot prefer to continue to live (over the millenia); its members can prefer to live (for the duration of their own time on earth). In some cases talking about a species is just metonymy, or shorthand, but in other cases I don't think that usage fits. For example, if "we must ensure the human race persists" is just shorthand, it's shorthand for "people should have children", and it's strange to insist that other persons have the personal desire to have children. When someone is suffering, e.g. from depression, we can encourage them to continue to live for hope of a time where they aren't suffering (by helping them imagine a plausible, happier future), which to them might be worth enduring their present circumstances. In other words, we want to help them make a better-informed decision about their own life or death by reminding them that their current outlook is strongly colored by their current mental state. But when it comes to a species, the future will not be experienced by any of its currently living members – it will be experienced by individuals who don't yet exist! And if they are never born, they've not been robbed of life or had a preference violated (because there's no one to rob or disregard). The philosophy is much more complicated than I'm presenting, but I wanted to say something on the matter, if only because it's the sort of questioning I wish I saw more often.

ianburrell 3y ago
One thing that annoys me about longtermism is that they focus on the survival of human race. I can't tell if this means humans as currently exist or humans as will exist in the future. Because humans will change a lot conquering the galaxy and surviving for a long span of time. How much change is allowed before considered non-human? Do all human descendants count? Does this include modifications or cyborgs? Or uploaded minds? Most importantly, they don't seem to think of AIs as human descendants. Is it better for human-like AIs to conquer the galaxy, and/or for humans to be trapped on Earth for a million years? If that is the only choice.