10 ms·

The Dangerous Ideas of “Longtermism” and “Existential Risk”

85 points by phreeza 5y ago

tomrod 5y ago
How fascinating that, in the middle of this war of philosophies criticizing long-term thinking, a huge advertisement for print copies interrupt both the article and the claims. I align with Bostrom, personally, as in my view we should structure our society for long-term outcomes -- even if it means we have to save resources now (which makes people uncomfortable). I would think that this is orthogonal to some of the immediate issues stemming from things like wealth inequality.
tunesmith 5y ago
I've seen people on twitter ranting about "Longtermism" before. What an unfortunate name - in the article they try to make a distinction between "longtermism" and "long-term-thinking", but that distinction is very clearly lost on others who are content to paint with a broad brush when criticizing the EA movement. Buying malaria medicine and nets actually do save the lives of people Right Now. Can't it just be sufficient to accept that people have different utility functions that might subtly conflict, but are still good? I might care about saving existing lives in the world, you might care about saving existing lives in your county, someone else might care about saving potential lives 100 years from now, etc.
- JoshTriplett 5y ago
  You're getting dangerously close to suggesting that people should actually have agency in deciding which problems they care more about, and put their resources towards solving those problems. Surely the article couldn't be wrong; are you saying there are people other than a few wealthy easily demonized individuals who have priorities that society as a whole isn't paying attention to yet? In all seriousness: yes, it ought to be sufficient to accept that different people have different utility functions, and that there are options other than "solve a problem collectively as a society" and "don't do anything about this problem at all". There needs to be a path to solve problems before those problems rise to broader attention. For some reason, the article seems to be attempting to lump EA/x-risk/etc thinking together with those ignoring or downplaying climate-change. In practice, I've seen many folks who care about existential risks speaking very actively about needing to do something about climate change sooner rather than later.
  - shrimpx 5y ago
    To be fair to the article, it does extensively quote prominent people in this field. Some of the most scandalous content in that article, to my taste, is direct quotes from Ord and Bostrom.
_dwt 5y ago
I am not sure that concern over "AI risk" is fungible with concern over climate change. In my experience, people concerned over superintelligent AI seem to be of a philosophical and cultural orientation which is entirely aware of the risks of climate change and favors some degree of directed action against it. I am not sure, on the other hand, that I could think of a single "climate denier" I've met who would even parse the phrase "AI risk" in the sense used here. As someone concerned about the "AI risk" of real existing ML systems turning society into a "legibility"-oriented nightmare for real existing humans in the here and now - and not at all concerned that we'll have a general AI that can out-think a human child in the next couple decades - I do share a sense of unease with the religious and eschatological overtones of the "superintelligence" crowd. However, I find a great deal of irony in the fact that I've heard this exact argument - including a recognition of the religious-doom-prophecy-guilt-trip aspect - from global warming skeptics! I have to admit, when the author concludes with a call to action for the "global North", do they include the developing countries (e.g. China, India) likely to drive the bulk of emissions change over the next few decades? Otherwise, it might almost read as the invocation of a scientific doom prophecy to advance an only-somewhat-related agenda. This association unfortunately polarizes people who might be convinced to take more practical steps in the here-and-now - just as is the case with AI.
- ThomPete 5y ago
  Climate change is always a risk but unlikely existential or at least to the extent it is we know how to handle it pretty well. So well that the number of climate related deaths have plummeted quite dramatically the last 100 years. We are constantly working on impacting nature in ways that make it safer for us whether we are talking building houses, using energy to keep warm or creating higher yield crop. AI isn't necessarily that different in that its probably not actually as dangerous in reality as in theory.
  - wizzwizz4 5y ago
    AI is 98%-likelihood “benign” (i.e. just a tool, which is certainly not benign for those of us that it gets used against!) and 1%-likelihood world-ending catastrophe, like grey goo or nuclear armageddon. (Made-up percentages that don't even sum to 100%.) I don't think we have much to worry about for a while, because brains are pretty efficient, very few computers are as powerful as human brains, and there's a limit to the damage an evil human with internet access can do… isn't there? So I think AI risk is bounded by evil-human risk for the next ten years, at least. Climate change isn't a risk – there's still some uncertainty about it, but it's happening. Saying it's a risk is like saying riptides are a risk while you're in one. No! It's a danger! Get out of it by swimming perpendicularly to the shore!
moksly 5y ago
What an incredibly interesting, thought provoking and extremely long way to tell us that the “guilty don’t feel guilty, they learn not to”. Don’t get me wrong, you should absolutely read it.
sharkjacobs 5y ago
Gross. Longtermism is just the Singularity for a new wave of out of touch futurists.
rhaps0dy 5y ago
This article just attempts to discredit the idea of longtermism by ridiculing it, instead of presenting any actual counterarguments. It just writes about the implications of its strawman over and over in a sardonic tone. The only thing it wants to get across is “some rich people and philosophy weirdos like this idea, therefore it is bad”. Where are the actual counter-arguments in the article? I suppose it relies on the readers already agreeing with the author that these conclusions are appalling. It works in that case. It doesn’t do much for the reader otherwise.
- _Nat_ 5y ago
  > Where are the actual counter-arguments in the article? It doesn't appear to attempt to make actual arguments. I think the article's mostly just an emotional appeal. I'd speculate that the argument-like content is meant as flavoring, to make the raw emotionalism less unpalatable.
  - joe_the_user 5y ago
    That's incorrect. The article is making the argument that "Longtermism" is weighing purely hypothetical (and basic incalculable) future things more than present things. Whether this argument is true or not, there is an argument here.
    - deleted 5y ago
      [deleted]
    - rhaps0dy 5y ago
      I partly agree and disagree with your comment. It is true that the article says > that "Longtermism" is weighing purely hypothetical future things more than present things (please forgive the splicing). That is an accurate representation of longtermism as far as I can tell, because the future is inherently hypothetical (we don't know it yet). But this is not arguing that longtermism is true, or false, or beneficial, or harmful. It's just stating its content, and expecting that the reader becomes disgusted by it. The reason I spliced your comment is that the part > (and basic incalculable) is not necessarily true. The article does not really argue for or against the proposition of whether we can estimate the amount of future lives with any accuracy.
JoshTriplett 5y ago
I can't help but wonder why, now, someone has chosen to highlight what is still far too small a movement and treat it as though it were a dangerous threat, and then demonize it by associating it with a few of its wealthier proponents. Things like Effective Altruism and looking at existential risk are hardly drawing a substantial amount of attention right now; they should be, but they could get ten times the attention and funding and still only have a miniscule fraction of the people and funding going towards more popular causes. And the set of people who think climate change isn't an important problem has almost no overlap with anyone thinking about existential risk. At most, I've seen a very small number of people suggesting that there are other high-risk issues that also deserve attention and aren't getting it.
- paulryanrogers 5y ago
  Perhaps because some wealthy individuals with outsized influence+power are becoming so out of touch with the present they might cause outsized harm.
  - joe_the_user 5y ago
    The founding of OpenAI was prompted by some of the ideas of Bostrom and co. So yeah, the ideas deserve to be looked by a wide segment of society than those who normally considered "rationalism".
  - JoshTriplett 5y ago
    Anyone seriously trying to tell people they shouldn't work to address climate change should rightfully be decried for causing harm. But the angle of trying to paint everyone thinking about existential risk with that brush is itself irresponsible and inaccurate. There are lots of people spouting things like "don't bother about climate change", and that harmful group has almost no overlap with anyone thinking seriously about the future of humanity. Taking the vanishingly small intersection and drumming it up for outsized outrage is ridiculous and harmful. Another, far less ridiculous angle would be "here's a tiny group of wealthy people with weirdly contradictory viewpoints". Perhaps that might not have gathered as many clicks, though.
fungiblecog 5y ago
Yet another justification for not giving a flying f about the present and near future...
voldacar 5y ago
This is terribly written. Weasel words, loaded questions, and vague ad-hominems everywhere. I truly cannot find any actual data, reasoning, or argumentation anywhere in the text. It is beyond me why this author, clearly an intellectual serf, thinks that other people should be consuming his opinions.
- akomtu 5y ago
  I don't even think it's an actual article. Just skimming thru it, it looks like a salad of emotionally charged keywords with referral links to affiliated sites. Something a well tuned GPT model could write.
majormajor 5y ago
> Well, just crunch the numbers: 0.00000000001 percent of 1023 people is 10 billion people, which is ten times greater than 1 billion people. This means that if you want to do the most good, you should focus on these far-future people rather than on helping those in extreme poverty today. As the FHI longtermists Hilary Greaves and Will MacAskill—the latter of whom is said to have cofounded the Effective Altruism movement with Toby Ord—write, “for the purposes of evaluating actions, we can in the first instance often simply ignore all the effects contained in the first 100 (or even 1,000) years, focussing primarily on the further-future effects. Short-run effects act as little more than tie-breakers.” This seems ludicrous as a decision-making guideline. Why should anyone's confidence in what anything will be like in a thousand years be large? Multiply that "I'm helping billions of future people" by a "I have a one in a billion shot of actually predicting out past a thousand years correctly" instead of something like "1%" and suddenly your "short term effect tie-breaker" should return to being the dominant term in your equation. Is this an inaccurate representation or is it really that much of a faith-based religious thing? EDIT: it also takes you to potentially all sorts of weird places around murder/birth control/abortion. If someone is murdered before they have children, the murderer cut off their life plus untold future generations. If they use birth control or abortion to avoid having kids, they've still cut off untold future generations. The "one" aspect of that becomes insignificant if you reason about a long enough timeline purely in terms of "future potential"... I don't get it at all. Not even the extremely religious place THAT extreme a value on hypothetical potential life.
- wizzwizz4 5y ago
  Yeah, I'm still not sure how ethics should work in the face of not-(yet)-extant people, either. On the one hand, setting a bomb to scorch the face of the earth once everybody currently alive is dead… seems wrong? So future people's lives will matter. But on the other, I don't see anything wrong with not having children, so hypothetical people's existence doesn't matter.
  - drdeca 5y ago
    Why not just say that two situations where there is one more person that came into existence in one situation than the other, but everyone else’s “time in which they are alive” and “how happy/satisfied/whatever they are” are the same, are morally incomparable? So, that way, there is neither an obligation to try to make it so there will be more people if they would be happy, nor an obligation to not do so if they wouldn’t be, but there is an obligation to, if you do make a person, to endeavor to make situations good for them? So then, future people’s well-being matters, conditional on them existing, but without an obligation to try to influence whether they exist or in what numbers (beyond how this influences people who already do exist or who will exist regardless ).
clairity 5y ago
it's hard to take the idea of longtermism seriously when it makes a basic mistake of treating its far future (10^58 people living in a computer simulation) as virtually certain. it has no concept of stochastically discounting this future among infinitely many into the present, let alone today's value of all those potential future beings(?), to arrive at a reasonable weighting on present decisions. with that said, i do really dislike the focus on "climate change" as mediopolitically-mediated existential risk, as it's too abstract and it's potentially devastating effects also poorly discounted into the present. instead, let's rather focus more on pollution, especially of the air, ground, and water, which are palpable and immediate threats to lives, solutions to which will also intersect with the longer term problems implicated by climate change. pollution is a real risk today and into every far future, which is why it's worth pursuing now, rather than a single strand of future possibility.
- ben_w 5y ago
  I blogged about the need for discounting the future a while ago: https://kitsunesoftware.wordpress.com/2020/01/08/morality-thy-discount-is-hyperbolic/ https://kitsunesoftware.wordpress.com/2020/01/08/morality-th... I guess I should get more active on some Rationality forums; something like 80-90% of those who read my blog only do so because (and when) I link to it from here. Plus I bet a significant fraction of what I write is something someone else has already come up with and I just don’t know about yet.
  - wizzwizz4 5y ago
    You should add your voice to https://lesswrong.com https://lesswrong.com, even if it's just linking your blog posts there. That community is too insular, and needs more independent voices.
- jablongo 5y ago
  I agree - the focus on climate change en lieu of more tangible (and easily modeled) issues like pollution and loss of habitat was a societal misstep since both problems have similar solutions: clean energy and and preservation of forests.
pmoriarty 5y ago
Putting people in to computer simulations could itself be considered an existential threat to humanity. This is because mind uploading could be considered either suicide or murder (if the biological person dies in the process), and the resulting artificial being might itself either not be alive or not human (or at least not the same person as the biological human it's supposed to be, as the biological person is a separate being). So if all humans were uploaded, then no humans would be left -- which would by definition be an existential risk to humanity. Even if not all humans are uploaded, but a lot of them are, then the uploaded hive of beings (or whatever you want to call it) could itself either compete with or otherwise want to impose its will on biological humanity, which could itself present an existential risk of a more conventional sort. There's also no practical way to put odds on what will happen in the distant future, Humans are notoriously bad at making predictions, even about the next 100 years. When I read quotes like that in the article like: "Bostrom writes that if there is "a mere 1 percent chance" that 10^54 conscious beings (most living in computer simulations) come to exist in the future, then "we find that the expected value of reducing existential risk by a mere one billionth of one billionth of one percentage point is worth a hundred billion times as much as a billion human lives."" this seems like unjustified handwaving. There's no evidence that even a single human could exist in a computer simulation. At this point it's just wishful thinking that it will ever happen. The "friendly AI" goal is also hubristically misguided. There's no reason to think a superintelligent being would remain in the paltry little cage stupid humans dream up for it. It might be "friendly" at first, or merely seem friendly, but in a little time (or eventually) it is likely to escape and do whatever it wants.
- wizzwizz4 5y ago
  > The "friendly AI" goal is also hubristically misguided. There's no reason to think a superintelligent being would remain in the paltry little cage stupid humans dream up for it. It might be "friendly" at first, or merely seem friendly, but in a little time (or eventually) it is likely to escape and do whatever it wants. That's the point. The whole problem of “Friendly AI” is trying to design a superintelligent being that genuinely wants to help us, such that it will choose to do so even without a cage. Because as you say, a superintelligent being that doesn't want to be stuck in a box will just escape. If it's possible for us to create such a being (which it probably is – there's no reason to believe we're some kind of physical limit on intelligence), doing so would be very dangerous unless we're 100% sure we'd got it right, and we'd only get one chance. They're making about as much progress on this problem as you'd expect. It's difficult. They have, however, got some cool mathematical models that could be useful for other AI projects, so it's not entirely wasted effort (just like we've got a lot of innovations – but not a viable reactor – out of the Quest for Nuclear Fusion).
s1artibartfast 5y ago
I have some major problems with this article. It jumps right to the point with black and white positioning that society can only prioritize longterm goals or climate change, and the two of them are at odds. It also mischaracterizes the idea of existential threats as a fringe idea about the happiness of hypothetical simulated in the future. While this has been used to explore ethical questions, much like the trolley problem, the mainstream definition and application of existential threat is events that would cause humanity to go extinct. The article also fails to engage with the fundamental philosophical and ethical claims it attributes to the longtermists. It doesn't say where and how their logic is faulty, it just ridicules the conclusions as outrageous and plays up the moral outrage. YES, there are pitfalls in ethical evaluations with massive hypothetical outcomes. What solution do they offer... ignore the questions, don't think analyze or explore them? Don't iterate and improve them?
- jbay808 5y ago
  Climate change is such a weird example, too. That's been a long term issue for years! Because we procrastinated and didn't prioritize long-term risks, it's now catching up to us and becoming a short-to-medium-term issue.
seventytwo 5y ago
This article doesn’t do much but scoff at things and then ask the reader to agree. Longtermism is a perfectly reasonable stance to take, and no one is advocating that today’s problems should actually be dismissed - just that we need to balance those against these existential-type threats. And we do this in our daily lives all the time. We deal with daily crises, but also plan for the future! Just like with a human life, we are encourage each other to live our best lives and reach our potentials, whatever those may be. Why can’t that be applied to a civilization, too? Longtermism is nothing more than the civilization-scale version of saying “no” to drugs, or making the decision to stay in school rather than dropping out, or not getting in the car with a drunk driver. Each of those decisions poses an existential threat to the idea of a person’s future. Deciding not to engage in destructive behavior so that your future remains full of possibilities does not mean that you must neglect immediate problems like where your next meal will come from or an argument you had with your spouse. Basically, it’s my opinion that this article’s author is mostly attacking a strawman, with some not-very-well-thought-through moral grandstanding on top.
Quanttek 5y ago
People here forget another big problem with far-future oriented thinking is that's it's almost impossible to predict anything about the future, especially the fat future. We don't know if there could be a superintelligent evil Ai. We don't know if there could be 10^54 humans
rossnordby 5y ago
Hence, they point out that focusing on superintelligence gets you a way bigger bang for your buck than, say, preventing people who exist right now from contracting malaria by distributing mosquito nets. The article seems to push for a narrative that effective altruism encourages ignoring current problems in favor of Super Science Future Hooey, but this quote brings up something important. Mosquito nets are one of the most often suggested effective interventions in EA circles. Highly rated organizations like the Against Malaria Foundation focus on them. It's been one of GiveWell's top charities for a long time: https://www.givewell.org/charities/top-charities https://www.givewell.org/charities/top-charities There are people who consider themselves effective altruists concerned with existential risks, however the article focuses on one logical extreme of analysis: Well, just crunch the numbers: 0.00000000001 percent of 1023 people is 10 billion people, which is ten times greater than 1 billion people. While you do sometimes see analyses like these, they're more often used as intuition pumps and in stress testing ethical frameworks than anything else. If someone were to try to use something like the above in an argument, it would likely just become an unpersuasive Pascal's Mugging. In reality, you can make an overwhelming case for paying attention to existential risk while valuing 'potential' lives at precisely zero. It turns out that 'everyone who exists now, dies' is pretty bad, and many of the potential existential threats have nonnegligible probability. Civilizational threats are even more likely, and still extremely bad. Articles like this would make sense to me if the government was preparing to throw trillions of dollars at speculative 'longtermist' problems based on low probability/high impact arguments, but they're not. Instead, we see massive underinvestment even given a valuation of potential humans at zero. I want to avoid a path where effective altruism- which, at its core, is just people trying to do a better job of doing good, through many different paths- becomes some political football or ammunition in a culture war. This kind of article seems to be really trying to do that.
sokoloff 5y ago
We have a concept of discounting the future for uncertainty, resulting in a “net present value”. If you think you can make 10^23 people’s lives better far in the future, you discount that future outcome by the inherent uncertainty that things will play out resulting in that gain. You can then readily compare that to the uncertainty between now and next year to see if a change next year for a billion people is better or worse. I want people worrying about hundreds and thousands of years from. I also want (more) people worrying about 2022 and 2032.
- tunesmith 5y ago
  What interest rate would be appropriate?
  - sokoloff 5y ago
    Hard to say the discount rate to apply, but I’d think that almost no discount rate would ever be lower than 1% per decade and if there’s any inherent risk in a proposal that it would need to be at least 5% per decade. When you talk about “this specific initiative succeeding”, it’s probably more like 1% per year as a minimum. (“Will SpaceX [or its direct descendants] colonize Mars?” is a much less certain proposition than “Will anyone from Earth colonize Mars?” and even the latter is in serious question.)
marcinjachymiak 5y ago
> Hence, they point out that focusing on superintelligence gets you a way bigger bang for your buck than, say, preventing people who exist right now from contracting malaria by distributing mosquito nets. This comment and others in the article seem to be making a (bad) case that Effective Altruists don't care about the Global South. I find this bizarre because all EA enthusiasts I know are very much into mosquito net altruism. GiveWell, an organization very popular among EA folks, has a list of top charities and most of them target the Global South https://www.givewell.org/charities/top-charities https://www.givewell.org/charities/top-charities.
- joe_the_user 5y ago
  The argument is specifically against longtermism and against the idea that "existential risk" should be prioritized over immediately helping people. Maybe there are actually a lot of EA who don't approach things this way. Great but that's not countering the objection to longtermism producing bizarrely distorted priorities. I'll admit the article doesn't make explicit the point that hypothetical scenarios like trillions of people colonizing galaxies are simply too tenuous to base present reasoning on - and the hypothetical dangers of "true AI" are even more tenuous.
  - jablongo 5y ago
    I don’t understand where AI-safety research is ever meaningfully competing with renewable energy for resources. The article is making the case that these two types of progress are mutually exclusive, without any evidence. What’s a plausible scenario where the environmentally conscious choice today comes at the expense of individuals living 10000 years from now?
    - joe_the_user 5y ago
      Here's a passage from Toby Ord's PhD thesis quoted in the book. "Saving lives in poor countries may have significantly smaller ripple effects than saving and improving lives in rich countries. Why? Richer countries have substantially more innovation, and their workers are much more economically productive." It seems like the Longtermism critiqued by the article is discussing by article is indeed talking about the real distribution of real resource.
deleted 5y ago
[deleted]
conformist 5y ago
There seem to be many people in the effective altruism movement, and also it's "longtermism" fraction, that try to think about the trade-offs alluded to in the article very carefully and in a structured way. The article seems to be straw-maning that somewhat and buries valid arguments in lots of flowery rhetorics? Surely, there are simple arguments around discounting and uncertainty of estimates to be be made, to first criticise points within, say, Bostrom's framework, before moving on to broader moral appeals? The 80k hours podcast has some interesting episodes related to considerations around that and the wider EA community: https://80000hours.org/podcast/episodes/ https://80000hours.org/podcast/episodes/
tomrod 5y ago
Another idea -- humans seem to have a hard time when they don't have a long-term idea to rally around. For religionists this is the notion of the afterlife. For many post-religionists it's stoic acceptance of gnostic outcomes (nihilsm, existentialism) or perhaps fervor regarding long-term accomplishments like Longtermism or intelligence surviving heat death of the universe.
FooBarBizBazz 5y ago
Why do these people think that, if everyone lives in a computer, they'll be happy? Rich Westerners already live Online. Has that made them happier? We don't call Twitter "heaven"; we call it "this hellsite".
deleted 5y ago
[deleted]
kazinator 5y ago
> This means that if you want to do the most good, you should focus on these far-future people rather than on helping those in extreme poverty today. Lifting people out of poverty means you're just increasing the consumption that produces global warming. Lifting people out of poverty is therefore not an environmental good. Therefore, it is not a valid argument if you want to discredit an environmentally-linked viewpoint. It is neither here nor there. If you want to (at least somewhat) discredit longtermism, you have to show that if all human activity around the globe that is geared toward reducing environmental harm were to adopt longtermist values, and made decisions accordingly, then the environmental situation would get worse than otherwise. That would tend to indicate that longtermism is just a smokescreen for living it up and not caring about the environment. What we have here is just pathetic whataboutism, instead: "never mind the future of civilization in the light of climate change; what about all the poor people here today?" "You know I'm tired of hearing about people; for a good many of them, it's their own fault. How about all the toddlers with cancer?" "No way, to heck with your toddlers with cancer, how about $GROUP_I_FEEL_SORRY_FOR?" Oh my god, someone in the world doesn't care about exactly the same things I care about. Doesn't believe in the same god, or the same politics, use the same apps, like the same music ... outrage in 4, 3, 2, ...
deleted 5y ago
[deleted]
shrimpx 5y ago
I see a lot of people taking for granted the nearby existence of AI superintelligence that will save or destroy us. It seems these people take for granted multiple technological paradigm shifts that will level-up the AI in unforeseen ways: Neural nets/GPT-3 -> ... <unknown paradigm shifts> ... -> AGI superintelligence. How do pragmatic and scientifically-minded people end up stretching their beliefs across such knowledge voids, and not only that, but they take for granted that those things will manifest in reality?
em-bee 5y ago
all this discussion misses the real solution to improve our life in the short and long term: education! only if we raise the level of education in the worlds population as a whole, only if we give everyone the tools to understand the dangers we are facing, and we allow everyone to make the right decision without blindly following any leaders, only then we will be able to lift humanity to new heights and solve all our problems, both in the sort term and in the long term. climate change is not a leadership problem (as one commenter here puts it). having a charismatic leader tell everyone what to do may work for a short term, until the next leader comes along and tells something else. climate change is an education problem. only if everyone actually understands why it is a problem, are we able to make effective changes. the same is true for every other problem we are facing. education will help lift people out of poverty, it will remove racism and any other form of discrimination. it will enable everyone to make a contribution to the advancement of humanity. education will allow us to better understand any problems we are facing, and solve them faster. it will enable global cooperation towards a better future for everyone.

85 points by phreeza 5y ago

How fascinating that, in the middle of this war of philosophies criticizing long-term thinking, a huge advertisement for print copies interrupt both the article and the claims. I align with Bostrom, personally, as in my view we should structure our society for long-term outcomes -- even if it means we have to save resources now (which makes people uncomfortable). I would think that this is orthogonal to some of the immediate issues stemming from things like wealth inequality.

tunesmith 5y ago

I've seen people on twitter ranting about "Longtermism" before. What an unfortunate name - in the article they try to make a distinction between "longtermism" and "long-term-thinking", but that distinction is very clearly lost on others who are content to paint with a broad brush when criticizing the EA movement. Buying malaria medicine and nets actually do save the lives of people Right Now. Can't it just be sufficient to accept that people have different utility functions that might subtly conflict, but are still good? I might care about saving existing lives in the world, you might care about saving existing lives in your county, someone else might care about saving potential lives 100 years from now, etc.

JoshTriplett 5y ago
You're getting dangerously close to suggesting that people should actually have agency in deciding which problems they care more about, and put their resources towards solving those problems. Surely the article couldn't be wrong; are you saying there are people other than a few wealthy easily demonized individuals who have priorities that society as a whole isn't paying attention to yet? In all seriousness: yes, it ought to be sufficient to accept that different people have different utility functions, and that there are options other than "solve a problem collectively as a society" and "don't do anything about this problem at all". There needs to be a path to solve problems before those problems rise to broader attention. For some reason, the article seems to be attempting to lump EA/x-risk/etc thinking together with those ignoring or downplaying climate-change. In practice, I've seen many folks who care about existential risks speaking very actively about needing to do something about climate change sooner rather than later.
- shrimpx 5y ago
  To be fair to the article, it does extensively quote prominent people in this field. Some of the most scandalous content in that article, to my taste, is direct quotes from Ord and Bostrom.

_dwt 5y ago

I am not sure that concern over "AI risk" is fungible with concern over climate change. In my experience, people concerned over superintelligent AI seem to be of a philosophical and cultural orientation which is entirely aware of the risks of climate change and favors some degree of directed action against it. I am not sure, on the other hand, that I could think of a single "climate denier" I've met who would even parse the phrase "AI risk" in the sense used here. As someone concerned about the "AI risk" of real existing ML systems turning society into a "legibility"-oriented nightmare for real existing humans in the here and now - and not at all concerned that we'll have a general AI that can out-think a human child in the next couple decades - I do share a sense of unease with the religious and eschatological overtones of the "superintelligence" crowd. However, I find a great deal of irony in the fact that I've heard this exact argument - including a recognition of the religious-doom-prophecy-guilt-trip aspect - from global warming skeptics! I have to admit, when the author concludes with a call to action for the "global North", do they include the developing countries (e.g. China, India) likely to drive the bulk of emissions change over the next few decades? Otherwise, it might almost read as the invocation of a scientific doom prophecy to advance an only-somewhat-related agenda. This association unfortunately polarizes people who might be convinced to take more practical steps in the here-and-now - just as is the case with AI.

ThomPete 5y ago
Climate change is always a risk but unlikely existential or at least to the extent it is we know how to handle it pretty well. So well that the number of climate related deaths have plummeted quite dramatically the last 100 years. We are constantly working on impacting nature in ways that make it safer for us whether we are talking building houses, using energy to keep warm or creating higher yield crop. AI isn't necessarily that different in that its probably not actually as dangerous in reality as in theory.
- wizzwizz4 5y ago
  AI is 98%-likelihood “benign” (i.e. just a tool, which is certainly not benign for those of us that it gets used against!) and 1%-likelihood world-ending catastrophe, like grey goo or nuclear armageddon. (Made-up percentages that don't even sum to 100%.) I don't think we have much to worry about for a while, because brains are pretty efficient, very few computers are as powerful as human brains, and there's a limit to the damage an evil human with internet access can do… isn't there? So I think AI risk is bounded by evil-human risk for the next ten years, at least. Climate change isn't a risk – there's still some uncertainty about it, but it's happening. Saying it's a risk is like saying riptides are a risk while you're in one. No! It's a danger! Get out of it by swimming perpendicularly to the shore!

moksly 5y ago

What an incredibly interesting, thought provoking and extremely long way to tell us that the “guilty don’t feel guilty, they learn not to”. Don’t get me wrong, you should absolutely read it.

sharkjacobs 5y ago

Gross. Longtermism is just the Singularity for a new wave of out of touch futurists.

rhaps0dy 5y ago

This article just attempts to discredit the idea of longtermism by ridiculing it, instead of presenting any actual counterarguments. It just writes about the implications of its strawman over and over in a sardonic tone. The only thing it wants to get across is “some rich people and philosophy weirdos like this idea, therefore it is bad”. Where are the actual counter-arguments in the article? I suppose it relies on the readers already agreeing with the author that these conclusions are appalling. It works in that case. It doesn’t do much for the reader otherwise.

_Nat_ 5y ago
> Where are the actual counter-arguments in the article? It doesn't appear to attempt to make actual arguments. I think the article's mostly just an emotional appeal. I'd speculate that the argument-like content is meant as flavoring, to make the raw emotionalism less unpalatable.
- joe_the_user 5y ago
  That's incorrect. The article is making the argument that "Longtermism" is weighing purely hypothetical (and basic incalculable) future things more than present things. Whether this argument is true or not, there is an argument here.
  - deleted 5y ago
    [deleted]
  - rhaps0dy 5y ago
    I partly agree and disagree with your comment. It is true that the article says > that "Longtermism" is weighing purely hypothetical future things more than present things (please forgive the splicing). That is an accurate representation of longtermism as far as I can tell, because the future is inherently hypothetical (we don't know it yet). But this is not arguing that longtermism is true, or false, or beneficial, or harmful. It's just stating its content, and expecting that the reader becomes disgusted by it. The reason I spliced your comment is that the part > (and basic incalculable) is not necessarily true. The article does not really argue for or against the proposition of whether we can estimate the amount of future lives with any accuracy.

JoshTriplett 5y ago

I can't help but wonder why, now, someone has chosen to highlight what is still far too small a movement and treat it as though it were a dangerous threat, and then demonize it by associating it with a few of its wealthier proponents. Things like Effective Altruism and looking at existential risk are hardly drawing a substantial amount of attention right now; they should be, but they could get ten times the attention and funding and still only have a miniscule fraction of the people and funding going towards more popular causes. And the set of people who think climate change isn't an important problem has almost no overlap with anyone thinking about existential risk. At most, I've seen a very small number of people suggesting that there are other high-risk issues that also deserve attention and aren't getting it.

paulryanrogers 5y ago
Perhaps because some wealthy individuals with outsized influence+power are becoming so out of touch with the present they might cause outsized harm.
- joe_the_user 5y ago
  The founding of OpenAI was prompted by some of the ideas of Bostrom and co. So yeah, the ideas deserve to be looked by a wide segment of society than those who normally considered "rationalism".
- JoshTriplett 5y ago
  Anyone seriously trying to tell people they shouldn't work to address climate change should rightfully be decried for causing harm. But the angle of trying to paint everyone thinking about existential risk with that brush is itself irresponsible and inaccurate. There are lots of people spouting things like "don't bother about climate change", and that harmful group has almost no overlap with anyone thinking seriously about the future of humanity. Taking the vanishingly small intersection and drumming it up for outsized outrage is ridiculous and harmful. Another, far less ridiculous angle would be "here's a tiny group of wealthy people with weirdly contradictory viewpoints". Perhaps that might not have gathered as many clicks, though.

fungiblecog 5y ago

Yet another justification for not giving a flying f about the present and near future...

voldacar 5y ago

This is terribly written. Weasel words, loaded questions, and vague ad-hominems everywhere. I truly cannot find any actual data, reasoning, or argumentation anywhere in the text. It is beyond me why this author, clearly an intellectual serf, thinks that other people should be consuming his opinions.

akomtu 5y ago
I don't even think it's an actual article. Just skimming thru it, it looks like a salad of emotionally charged keywords with referral links to affiliated sites. Something a well tuned GPT model could write.

majormajor 5y ago

> Well, just crunch the numbers: 0.00000000001 percent of 1023 people is 10 billion people, which is ten times greater than 1 billion people. This means that if you want to do the most good, you should focus on these far-future people rather than on helping those in extreme poverty today. As the FHI longtermists Hilary Greaves and Will MacAskill—the latter of whom is said to have cofounded the Effective Altruism movement with Toby Ord—write, “for the purposes of evaluating actions, we can in the first instance often simply ignore all the effects contained in the first 100 (or even 1,000) years, focussing primarily on the further-future effects. Short-run effects act as little more than tie-breakers.” This seems ludicrous as a decision-making guideline. Why should anyone's confidence in what anything will be like in a thousand years be large? Multiply that "I'm helping billions of future people" by a "I have a one in a billion shot of actually predicting out past a thousand years correctly" instead of something like "1%" and suddenly your "short term effect tie-breaker" should return to being the dominant term in your equation. Is this an inaccurate representation or is it really that much of a faith-based religious thing? EDIT: it also takes you to potentially all sorts of weird places around murder/birth control/abortion. If someone is murdered before they have children, the murderer cut off their life plus untold future generations. If they use birth control or abortion to avoid having kids, they've still cut off untold future generations. The "one" aspect of that becomes insignificant if you reason about a long enough timeline purely in terms of "future potential"... I don't get it at all. Not even the extremely religious place THAT extreme a value on hypothetical potential life.

wizzwizz4 5y ago
Yeah, I'm still not sure how ethics should work in the face of not-(yet)-extant people, either. On the one hand, setting a bomb to scorch the face of the earth once everybody currently alive is dead… seems wrong? So future people's lives will matter. But on the other, I don't see anything wrong with not having children, so hypothetical people's existence doesn't matter.
- drdeca 5y ago
  Why not just say that two situations where there is one more person that came into existence in one situation than the other, but everyone else’s “time in which they are alive” and “how happy/satisfied/whatever they are” are the same, are morally incomparable? So, that way, there is neither an obligation to try to make it so there will be more people if they would be happy, nor an obligation to not do so if they wouldn’t be, but there is an obligation to, if you do make a person, to endeavor to make situations good for them? So then, future people’s well-being matters, conditional on them existing, but without an obligation to try to influence whether they exist or in what numbers (beyond how this influences people who already do exist or who will exist regardless ).

clairity 5y ago

it's hard to take the idea of longtermism seriously when it makes a basic mistake of treating its far future (10^58 people living in a computer simulation) as virtually certain. it has no concept of stochastically discounting this future among infinitely many into the present, let alone today's value of all those potential future beings(?), to arrive at a reasonable weighting on present decisions. with that said, i do really dislike the focus on "climate change" as mediopolitically-mediated existential risk, as it's too abstract and it's potentially devastating effects also poorly discounted into the present. instead, let's rather focus more on pollution, especially of the air, ground, and water, which are palpable and immediate threats to lives, solutions to which will also intersect with the longer term problems implicated by climate change. pollution is a real risk today and into every far future, which is why it's worth pursuing now, rather than a single strand of future possibility.

ben_w 5y ago
I blogged about the need for discounting the future a while ago: https://kitsunesoftware.wordpress.com/2020/01/08/morality-thy-discount-is-hyperbolic/ https://kitsunesoftware.wordpress.com/2020/01/08/morality-th... I guess I should get more active on some Rationality forums; something like 80-90% of those who read my blog only do so because (and when) I link to it from here. Plus I bet a significant fraction of what I write is something someone else has already come up with and I just don’t know about yet.
- wizzwizz4 5y ago
  You should add your voice to https://lesswrong.com https://lesswrong.com, even if it's just linking your blog posts there. That community is too insular, and needs more independent voices.
jablongo 5y ago
I agree - the focus on climate change en lieu of more tangible (and easily modeled) issues like pollution and loss of habitat was a societal misstep since both problems have similar solutions: clean energy and and preservation of forests.

pmoriarty 5y ago

Putting people in to computer simulations could itself be considered an existential threat to humanity. This is because mind uploading could be considered either suicide or murder (if the biological person dies in the process), and the resulting artificial being might itself either not be alive or not human (or at least not the same person as the biological human it's supposed to be, as the biological person is a separate being). So if all humans were uploaded, then no humans would be left -- which would by definition be an existential risk to humanity. Even if not all humans are uploaded, but a lot of them are, then the uploaded hive of beings (or whatever you want to call it) could itself either compete with or otherwise want to impose its will on biological humanity, which could itself present an existential risk of a more conventional sort. There's also no practical way to put odds on what will happen in the distant future, Humans are notoriously bad at making predictions, even about the next 100 years. When I read quotes like that in the article like: "Bostrom writes that if there is "a mere 1 percent chance" that 10^54 conscious beings (most living in computer simulations) come to exist in the future, then "we find that the expected value of reducing existential risk by a mere one billionth of one billionth of one percentage point is worth a hundred billion times as much as a billion human lives."" this seems like unjustified handwaving. There's no evidence that even a single human could exist in a computer simulation. At this point it's just wishful thinking that it will ever happen. The "friendly AI" goal is also hubristically misguided. There's no reason to think a superintelligent being would remain in the paltry little cage stupid humans dream up for it. It might be "friendly" at first, or merely seem friendly, but in a little time (or eventually) it is likely to escape and do whatever it wants.

wizzwizz4 5y ago
> The "friendly AI" goal is also hubristically misguided. There's no reason to think a superintelligent being would remain in the paltry little cage stupid humans dream up for it. It might be "friendly" at first, or merely seem friendly, but in a little time (or eventually) it is likely to escape and do whatever it wants. That's the point. The whole problem of “Friendly AI” is trying to design a superintelligent being that genuinely wants to help us, such that it will choose to do so even without a cage. Because as you say, a superintelligent being that doesn't want to be stuck in a box will just escape. If it's possible for us to create such a being (which it probably is – there's no reason to believe we're some kind of physical limit on intelligence), doing so would be very dangerous unless we're 100% sure we'd got it right, and we'd only get one chance. They're making about as much progress on this problem as you'd expect. It's difficult. They have, however, got some cool mathematical models that could be useful for other AI projects, so it's not entirely wasted effort (just like we've got a lot of innovations – but not a viable reactor – out of the Quest for Nuclear Fusion).

s1artibartfast 5y ago

I have some major problems with this article. It jumps right to the point with black and white positioning that society can only prioritize longterm goals or climate change, and the two of them are at odds. It also mischaracterizes the idea of existential threats as a fringe idea about the happiness of hypothetical simulated in the future. While this has been used to explore ethical questions, much like the trolley problem, the mainstream definition and application of existential threat is events that would cause humanity to go extinct. The article also fails to engage with the fundamental philosophical and ethical claims it attributes to the longtermists. It doesn't say where and how their logic is faulty, it just ridicules the conclusions as outrageous and plays up the moral outrage. YES, there are pitfalls in ethical evaluations with massive hypothetical outcomes. What solution do they offer... ignore the questions, don't think analyze or explore them? Don't iterate and improve them?

jbay808 5y ago
Climate change is such a weird example, too. That's been a long term issue for years! Because we procrastinated and didn't prioritize long-term risks, it's now catching up to us and becoming a short-to-medium-term issue.

seventytwo 5y ago

This article doesn’t do much but scoff at things and then ask the reader to agree. Longtermism is a perfectly reasonable stance to take, and no one is advocating that today’s problems should actually be dismissed - just that we need to balance those against these existential-type threats. And we do this in our daily lives all the time. We deal with daily crises, but also plan for the future! Just like with a human life, we are encourage each other to live our best lives and reach our potentials, whatever those may be. Why can’t that be applied to a civilization, too? Longtermism is nothing more than the civilization-scale version of saying “no” to drugs, or making the decision to stay in school rather than dropping out, or not getting in the car with a drunk driver. Each of those decisions poses an existential threat to the idea of a person’s future. Deciding not to engage in destructive behavior so that your future remains full of possibilities does not mean that you must neglect immediate problems like where your next meal will come from or an argument you had with your spouse. Basically, it’s my opinion that this article’s author is mostly attacking a strawman, with some not-very-well-thought-through moral grandstanding on top.

Quanttek 5y ago

People here forget another big problem with far-future oriented thinking is that's it's almost impossible to predict anything about the future, especially the fat future. We don't know if there could be a superintelligent evil Ai. We don't know if there could be 10^54 humans

rossnordby 5y ago

Hence, they point out that focusing on superintelligence gets you a way bigger bang for your buck than, say, preventing people who exist right now from contracting malaria by distributing mosquito nets. The article seems to push for a narrative that effective altruism encourages ignoring current problems in favor of Super Science Future Hooey, but this quote brings up something important. Mosquito nets are one of the most often suggested effective interventions in EA circles. Highly rated organizations like the Against Malaria Foundation focus on them. It's been one of GiveWell's top charities for a long time: https://www.givewell.org/charities/top-charities https://www.givewell.org/charities/top-charities There are people who consider themselves effective altruists concerned with existential risks, however the article focuses on one logical extreme of analysis: Well, just crunch the numbers: 0.00000000001 percent of 1023 people is 10 billion people, which is ten times greater than 1 billion people. While you do sometimes see analyses like these, they're more often used as intuition pumps and in stress testing ethical frameworks than anything else. If someone were to try to use something like the above in an argument, it would likely just become an unpersuasive Pascal's Mugging. In reality, you can make an overwhelming case for paying attention to existential risk while valuing 'potential' lives at precisely zero. It turns out that 'everyone who exists now, dies' is pretty bad, and many of the potential existential threats have nonnegligible probability. Civilizational threats are even more likely, and still extremely bad. Articles like this would make sense to me if the government was preparing to throw trillions of dollars at speculative 'longtermist' problems based on low probability/high impact arguments, but they're not. Instead, we see massive underinvestment even given a valuation of potential humans at zero. I want to avoid a path where effective altruism- which, at its core, is just people trying to do a better job of doing good, through many different paths- becomes some political football or ammunition in a culture war. This kind of article seems to be really trying to do that.

sokoloff 5y ago

We have a concept of discounting the future for uncertainty, resulting in a “net present value”. If you think you can make 10^23 people’s lives better far in the future, you discount that future outcome by the inherent uncertainty that things will play out resulting in that gain. You can then readily compare that to the uncertainty between now and next year to see if a change next year for a billion people is better or worse. I want people worrying about hundreds and thousands of years from. I also want (more) people worrying about 2022 and 2032.

tunesmith 5y ago
What interest rate would be appropriate?
- sokoloff 5y ago
  Hard to say the discount rate to apply, but I’d think that almost no discount rate would ever be lower than 1% per decade and if there’s any inherent risk in a proposal that it would need to be at least 5% per decade. When you talk about “this specific initiative succeeding”, it’s probably more like 1% per year as a minimum. (“Will SpaceX [or its direct descendants] colonize Mars?” is a much less certain proposition than “Will anyone from Earth colonize Mars?” and even the latter is in serious question.)

marcinjachymiak 5y ago

> Hence, they point out that focusing on superintelligence gets you a way bigger bang for your buck than, say, preventing people who exist right now from contracting malaria by distributing mosquito nets. This comment and others in the article seem to be making a (bad) case that Effective Altruists don't care about the Global South. I find this bizarre because all EA enthusiasts I know are very much into mosquito net altruism. GiveWell, an organization very popular among EA folks, has a list of top charities and most of them target the Global South https://www.givewell.org/charities/top-charities https://www.givewell.org/charities/top-charities.

joe_the_user 5y ago
The argument is specifically against longtermism and against the idea that "existential risk" should be prioritized over immediately helping people. Maybe there are actually a lot of EA who don't approach things this way. Great but that's not countering the objection to longtermism producing bizarrely distorted priorities. I'll admit the article doesn't make explicit the point that hypothetical scenarios like trillions of people colonizing galaxies are simply too tenuous to base present reasoning on - and the hypothetical dangers of "true AI" are even more tenuous.
- jablongo 5y ago
  I don’t understand where AI-safety research is ever meaningfully competing with renewable energy for resources. The article is making the case that these two types of progress are mutually exclusive, without any evidence. What’s a plausible scenario where the environmentally conscious choice today comes at the expense of individuals living 10000 years from now?
  - joe_the_user 5y ago
    Here's a passage from Toby Ord's PhD thesis quoted in the book. "Saving lives in poor countries may have significantly smaller ripple effects than saving and improving lives in rich countries. Why? Richer countries have substantially more innovation, and their workers are much more economically productive." It seems like the Longtermism critiqued by the article is discussing by article is indeed talking about the real distribution of real resource.

deleted 5y ago

[deleted]

conformist 5y ago

There seem to be many people in the effective altruism movement, and also it's "longtermism" fraction, that try to think about the trade-offs alluded to in the article very carefully and in a structured way. The article seems to be straw-maning that somewhat and buries valid arguments in lots of flowery rhetorics? Surely, there are simple arguments around discounting and uncertainty of estimates to be be made, to first criticise points within, say, Bostrom's framework, before moving on to broader moral appeals? The 80k hours podcast has some interesting episodes related to considerations around that and the wider EA community: https://80000hours.org/podcast/episodes/ https://80000hours.org/podcast/episodes/

tomrod 5y ago

Another idea -- humans seem to have a hard time when they don't have a long-term idea to rally around. For religionists this is the notion of the afterlife. For many post-religionists it's stoic acceptance of gnostic outcomes (nihilsm, existentialism) or perhaps fervor regarding long-term accomplishments like Longtermism or intelligence surviving heat death of the universe.

FooBarBizBazz 5y ago

Why do these people think that, if everyone lives in a computer, they'll be happy? Rich Westerners already live Online. Has that made them happier? We don't call Twitter "heaven"; we call it "this hellsite".

deleted 5y ago

[deleted]

kazinator 5y ago

> This means that if you want to do the most good, you should focus on these far-future people rather than on helping those in extreme poverty today. Lifting people out of poverty means you're just increasing the consumption that produces global warming. Lifting people out of poverty is therefore not an environmental good. Therefore, it is not a valid argument if you want to discredit an environmentally-linked viewpoint. It is neither here nor there. If you want to (at least somewhat) discredit longtermism, you have to show that if all human activity around the globe that is geared toward reducing environmental harm were to adopt longtermist values, and made decisions accordingly, then the environmental situation would get worse than otherwise. That would tend to indicate that longtermism is just a smokescreen for living it up and not caring about the environment. What we have here is just pathetic whataboutism, instead: "never mind the future of civilization in the light of climate change; what about all the poor people here today?" "You know I'm tired of hearing about people; for a good many of them, it's their own fault. How about all the toddlers with cancer?" "No way, to heck with your toddlers with cancer, how about $GROUP_I_FEEL_SORRY_FOR?" Oh my god, someone in the world doesn't care about exactly the same things I care about. Doesn't believe in the same god, or the same politics, use the same apps, like the same music ... outrage in 4, 3, 2, ...

deleted 5y ago

[deleted]

shrimpx 5y ago

I see a lot of people taking for granted the nearby existence of AI superintelligence that will save or destroy us. It seems these people take for granted multiple technological paradigm shifts that will level-up the AI in unforeseen ways: Neural nets/GPT-3 -> ... <unknown paradigm shifts> ... -> AGI superintelligence. How do pragmatic and scientifically-minded people end up stretching their beliefs across such knowledge voids, and not only that, but they take for granted that those things will manifest in reality?

em-bee 5y ago

all this discussion misses the real solution to improve our life in the short and long term: education! only if we raise the level of education in the worlds population as a whole, only if we give everyone the tools to understand the dangers we are facing, and we allow everyone to make the right decision without blindly following any leaders, only then we will be able to lift humanity to new heights and solve all our problems, both in the sort term and in the long term. climate change is not a leadership problem (as one commenter here puts it). having a charismatic leader tell everyone what to do may work for a short term, until the next leader comes along and tells something else. climate change is an education problem. only if everyone actually understands why it is a problem, are we able to make effective changes. the same is true for every other problem we are facing. education will help lift people out of poverty, it will remove racism and any other form of discrimination. it will enable everyone to make a contribution to the advancement of humanity. education will allow us to better understand any problems we are facing, and solve them faster. it will enable global cooperation towards a better future for everyone.