23 ms·

AI safety is an apocalypse cult and we should recognize that everyone who has ever been in an apocalypse cult thinks that "no, this time it's really the end of

AI safety is an apocalypse cult and we should recognize that everyone who has ever been in an apocalypse cult thinks that "no, this time it's really the end of the world". Everyone who believes crazy shit thinks they're being reasonable, so just because we're super sure we're being reasonable doesn't mean what we believe isn't crazy. Do we believe that we alone, of all people who have believed in the coming end, have it right this time?

JamesBarney 4y ago
There have been people for a long time who think the world is going to end. We can use your argument to dismiss anyone who fears any existential risk like asteroids, nuclear war, climate change, AI, etc... And since the world exists every group will be wrong except one. I think the risk of human extermination is much lower than many in this group believe because I think AI is much more compute bound than algorithm bound. So I think a hard takeoff is very unlikely. But while the median case for climate change is probably worse than the median case for AI. The existential risks for AI seem much higher. And it's a concern worth taking seriously.
- esperent 4y ago
  The issue is that the effective altruism crowd are focused exclusively on AI safety, at least in what they publish online. If they talked about all possible threats to humanity and how to solve them, from AI, to climate change, to nuclear bombs, to asteroids, to disease, they would be an interesting and important group. They don't. They hardly talk about anything else except AI. From the outside it looks a lot like an obsession. Personally I think they treat AI as an existential threat above all others because they consider themselves to be the pinnacle of intelligence in the universe and AI is showing that human intelligence might not be so special after all. So it's an existential threat to their sense of self. Other existential threats merely threaten their lives and are therefore not as important. This would be fine, if acknowledged with a degree of self awareness. But it's not, instead it's wrapped up in so many layers of convoluted logic that most people, insiders and outsiders both, can't see the real reason for this obsession.
  - rhn_mk1 4y ago
    You might want to get familiar with what effective altruism actually focuses on. https://www.effectivealtruism.org/articles/introduction-to-effective-altruism#what-are-some-examples-of-effective-altruism-in-practice https://www.effectivealtruism.org/articles/introduction-to-e... https://www.lesswrong.com/posts/JmmA2Mf5GrY9D6nQD/four-focus-areas-of-effective-altruism https://www.lesswrong.com/posts/JmmA2Mf5GrY9D6nQD/four-focus... https://www.givingwhatwecan.org/best-charities-to-donate-to-2023#donate-to-expertmanaged-funds https://www.givingwhatwecan.org/best-charities-to-donate-to-...
    - deleted 4y ago
      [deleted]
  - mitthrowaway2 4y ago
    I think they're worried about it because they honestly do care about existential risk. If you can come up with some more likely existential threats than AI, I'm sure they'll worry about that too.
  - yanderekko 4y ago
    >The issue is that the effective altruism crowd are focused exclusively on AI safety. That's really not true. There's a huge overlap in these groups, but the top EA charities are not shoveling all their money at AI alignment orgs... even if one can argue that they should.
    - whatshisface 4y ago
      Arguing that AI safety is needed only demonstrates that it's altruistic, but you have a lot farther to go if you want to say funding research in this decade can be effective.
  - Natsu 4y ago
    > The issue is that the effective altruism crowd are focused exclusively on AI safety. How many EA people do you know? That seems to get the most clicks, but I usually associate them with things like mosquito bed nets to help prevent malaria, because that's a far more common topic in my experience. I'm sure you can find particular people who are all about the AI risk stuff that seems pretty wild to me, too. But when you say "focused exclusively" I feel like you may have mostly seen news articles about wild AI stuff and not the average EA enthusiast who spends their time talking about picking efficient charities.
- zarzavat 4y ago
  > AI is much more compute bound than algorithm bound Any reason to believe this? The human brain runs on 20 watts (!). Obviously biological hardware is not directly comparable to silicon, perhaps silicon is intrinsically less efficient by some factor such that in order to obtain 20W of human brain compute we need to expend K times more in silicon. But is that factor really so large that we are compute-bound? How big would K have to be for us to be compute-bound anyway? If you could have Einstein for 200kW (i.e. silicon is intrinsically 1000x less efficient than brains) would that be compute-bound? It seems much more likely that we are algorithm-bound and simply not using our available compute efficiently.
  - pas 4y ago
    So there are many failure modes possible here. Yudkowsky's hard take off mentions "solving protein folding", nanomachines, persuading everyone to let them connect to the internet, infinite adaptability, etc. I think those are unrealistic, but obviously they still have a non-zero chance. More probably bad possibilities are AI-Hitler (monomaniacal populist absolute leader) or AI-Stalin (manipulative, smart, absolutely paranoid, rising through the ranks) ... so something that's human-like enough to be able to connect with humans, to manipulate them, but at the same time less affected by the psychological shortcomings. (Ie. such an AI could spend enough time to cross-interrogate every underling, constantly watch them, etc.) And yes, a very efficient immortal dictator is very bad news, but still bound by human-like limits. And the big infinite dollar question is could this hypothetical AI improve on itself by transcending human limits? Let's say by directly writing programs that it has conscious control over? Can it truly "watch" a 1000 video streams in real-time? Can it increase the number of its input-and-output channels while maintaining its human-like efficiency? Because it's very different to run a fast neural network that spits out a myriad labels for every frame of a video stream (YOLO does this already, but it's not 20W!) and to integrate those labels into actions based on a constantly evolving strategy. Sure maybe the hypothetical AI will simply run a lot of AlphaZero-like hybrid tree-search estimator-evaluator things ... Anyway, what I'm trying to say is that our 20W efficiency comes with getting tried very fast, and using "fast mode" thinking for everything. (Slow mode is the exception, and using it is so rare that we basically pop open a glass of champagne every time.)
- paulcole 4y ago
  > The existential risks for AI seem much higher What’s the upper bound for climate change’s existential risks? The end of human existence and society as we know it except for a relatively small number of survivors living in a world those of us alive today can barely imagine?
  - fidgewidge 4y ago
    There is no realistic scenario provided for by current scientific knowledge that would lead to that outcome. Waterworld was a film, not a textbook.
    - mitthrowaway2 4y ago
      The scenario described in this article comes the closest, detailing a mechanism that may have been responsible for the end-Permian mass extinction, wherein warming oceans become anoxic and begin to release huge quantities of hydrogen sulfide gas. The gas is directly lethal to most life in the oceans and on land, and destroys the ozone layer as a bonus. It's hard for me to imagine any way in which an agricultural human civilization could survive that scenario. https://www.scientificamerican.com/article/impact-from-the-deep/ https://www.scientificamerican.com/article/impact-from-the-d...
      - fidgewidge 4y ago
        By "climate change" people mean the sort of change that will be realistically caused by humans within the next few hundred years, not the sort of extinction caused by massive volcano activity that takes 10,000 years and happens once every 100 million years. Obviously if the climate changes drastically enough fast enough then everyone dies but nobody is suggesting that's the case for preset day human activity.
        mitthrowaway2 4y ago
        In terms of timeline, the key sentence would be "The so-called thermal extinction at the end of the Paleocene began when atmospheric CO2 was just under 1,000 parts per million (ppm)". The end-Permian was likely around 2500 ppm[1]. It doesn't really matter whether that carbon comes from supervolcanoes or from human emissions. When the article was written, CO2 concentration was 385 ppm. Today it's 421 ppm. If CO2 concentrations were to rise linearly, the timeline for reaching 1,000 ppm would be 250 years. Achieving that would require emissions to stabilize at ~2014 levels. If emissions keep rising, then perhaps it's closer to 150 years. If strong feedbacks kick in, like methane from melting permafrost, then maybe 100 years or less, or maybe we'll reach that happy 2500 ppm mark and see what a real mass-extinction looks like. Of course the ocean has a big thermal mass, so it will probably take quite some time to heat up enough to trigger the disaster even after we reach that level of atmospheric carbon. Hopefully everything will be fine? [1] https://www.nature.com/articles/s41467-021-22298-7 https://www.nature.com/articles/s41467-021-22298-7
- freejazz 4y ago
  Numbers used to justify ignoring common sense
- scotty79 4y ago
  > And since the world exists every group will be wrong except one. Except at most one.
- ElfinTrousers 4y ago
  Asteroids, nuclear war, climate change, and AI. As the old song says "One of these things is not like the other". We know that asteroids exist and many large ones are uncomfortably close to our home. We know that nuclear weapons exist and have seen them used. We know that the climate is getting hotter (*). AI is...well, people losing their shit over ChatGPT (**) aside, AI is not going to be real enough to worry about for a few more decades at least. (*) Anyone who's about to regurgitate some fossil-fuel industry talking points in response, just save your breath. (**) Above note goes double for this.
  - JohnPrine 4y ago
    Would you rather we wait until AIs are actually posing a threat for us to study ways to align them with human values? Tons of money already goes into fighting climate change and basically everyone on earth is aware of the threat it poses. The AI safety field is only about a decade old and is relatively unknown. Of course it makes sense to raise awareness there
    - ElfinTrousers 4y ago
      > Would you rather we wait until AIs are actually posing a threat for us to study ways to align them with human values? No, I would rather we wait until we are close enough to AI that we are talking about something concrete, rather than making wild speculations.
      - JohnPrine 4y ago
        Interesting. In your opinion, how should we decide when we're close enough that we're talking about something concrete?
        ElfinTrousers 4y ago
        Please see second asterisked point above.
voltaireodactyl 4y ago
On the other hand, we and we alone live in a time where a grapefruit-sized weapon could obliterate a reasonably sized town, AND when such weapons are increasingly in reach of non-state entities. I would imagine the chance of apocalypse rises as fewer people in agreement would be required to enact it.
deleted 4y ago
[deleted]
rvz 4y ago
I don't think there is anything safe about ChatGPT regurgitating and hallucinating false information about someone's business based on real legal proceedings that it had been trained on. [0] As long as it cannot also explain its own decisions transparently [1], then there there will always be a need for AI safety. [0] https://news.ycombinator.com/item?id=35063005 https://news.ycombinator.com/item?id=35063005 [1] https://news.ycombinator.com/item?id=35060285 https://news.ycombinator.com/item?id=35060285
thorum 4y ago
While you make a reasonable point in terms of human psychology, when you create an AI, you are leaving the realm of human psychology and dealing with something new that doesn’t have to play by the same rules. I’d argue that’s a big difference. We’re creating a fundamentally alien intelligence from scratch and basically just crossing our fingers and hoping to hell it cares about keeping us alive.
fshbbdssbbgdd 4y ago
Personally, I find the AI safety arguments sound pretty far-fetched, but I can’t come up with solid arguments against them. Also, when I look for a good ”debunking”, I don’t find anything. Mostly all I find are mockery, ad-hominem and the like. Frankly, the people who argue climate change isn’t real do a better job than the people who argue AI risk isn’t real, and they’re wrong!
- j_crick 4y ago
  > but I can’t come up with solid arguments against them. Let chaos take the world! /s
- Vecr 4y ago
  Yeah, I can't really argue against the whole idea. My "P(doom)"(language from the article), at least as related to agentic AI, is pretty low. About 7 or 8 percent before I die, and I'm not really old. There's probably not a lot I could do about it anyway, given that most AI jobs are alleged to just make it worse. That said, I can't really go around telling people with different numbers they should be working on antiviral research or physical manufacturing, just because I (personally, without their massive and cohesive "evidence" (i.e. gamed out scenarios) bases) think that's more important.
  - vcxy 4y ago
    "Roll a D20 and if you get 1 humanity as you know it will be destroyed in your lifetime" seems bad. Why do you consider 7 or 8 percent low?
    - Vecr 4y ago
      I'm not one of those "all value in the universe" people, P(doom) for me does not have to be that bad. I would actually consider the inability to manufacture or unwillingness to manufacture certain things at scale in the US to count. Generally, my point is that agentic AI is not something I can really do much about, and other things have a combined P(doom) probability higher than it. The unwillingness in certain groups to use what they consider to be toxic or "forever" chemicals in the manufacture of physical products at scale that can make them world a better and less fragile place is something that could be worked in instead.
yanderekko 4y ago
"Apocalypse cults are always wrong, AI safety is an apocalypse cult, therefore it is (probably) wrong" is an argument which really only betrays the ignorance of the speaker regarding the specifics of the AI doomer argument. Why are apocalypse cults always wrong? Do the same specific reasons apply to the doomers here?
- geraldhh 4y ago
  > Why are apocalypse cults always wrong? because we are still here
  - yanderekko 4y ago
    That's the consequence of them being wrong, not the cause..
    - geraldhh 4y ago
      denying the existence of circular dependencies does not invalidate them
  - roenxi 4y ago
    If there was an apocalypse cult in ancient Pompeii predicting the mountain would explode and bury everyone with ash, they would not be here. And there easily could have been. This is not an argument, it is a tautology powered by the anthropic principle.
    - geraldhh 4y ago
      that example is a local disaster, not an apocalypse. but yes, there most certainly were people predicting it and they probably didn't get a chance to enjoy being right.
- simple-thoughts 4y ago
  Humans are attracted to certain types of mythological stories - and there’s nothing wrong with that. It becomes a problem when the stories and reality are conflated, like is happening with AI. The idea that apocalypse cults are always wrong comes from two source: (1) empirical evidence, every time there’s been one it’s wrong and (2) understanding human sociology, humans are susceptible to making certain types of thought errors as groups and apocalypse cults such as the AI one demonstrate those errors everywhere. So yeah an apocalypse cult could be right, but it would necessarily be for the wrong reasons. Just like a broken clock that matches the current time 2 minutes of the full day.
precompute 4y ago
IMO AI Safety is to AI as HR is to Employees. You can't just learn it out of a book and even then it's a pretty bad experience for everyone involved. The world is not a corporation and the existence of "AI Safety" just guarantees that no "mere mortals" will ever get to touch AI. Waiting for the day our Overlords make Graphic Card 2.0 and refuse to sell any to the plebs 'cause the overlords "know better"
kelnage 4y ago
Absolutely agree. My biggest problem with AI safety is that, simply, the problem they envisage doesn't exist yet (generally, at a minimum, relying on the existence of "AGI"). Hence discussions about it have to make a huge amount of assumptions about a whole range of aspects of what the AI threat will be - what the AI will be capable of, what it's impact will be - before getting on to what possible solutions might be relevant to preventing it. But given the first two are so undefined, the later is pure speculation - one that is difficult to criticise directly, because any specific critisms can usually be easy deflected by adjusting any of the above assumptions without making a substantial change to the "inevitable" conclusion. That's why it feels like an apocalypse cult to me - it's a conclusion, that has little strong evidence today, stacked on top of a constantly shifting set of assumptions, allowing adherents to avoid backing their arguments with evidence.
- shapefrog 4y ago
  For the freseable future the real danger I see from AI is inferior basic statistical models being jammed into production pretending to be some sort of all seeing all knowing AGI - a class of product we are not really that close to.
hiAndrewQuinn 4y ago
I don't have any solid arguments either, but I just get the feeling that after the first very public tragedy, like a killbot rampaging in New York City, we'll see a very, very sudden tilt in the public eye towards never wanting to touch this technology again. People are terrified of killer robots. One of the likely outcomes like that is even stronger pivoting towards walled gardens and end to end stuff than we previously had on the Internet. Can't train a model on someone else's encrypted database without decrypting it first.
- CatWChainsaw 4y ago
  Thou shalt not make a machine in the likeness of a man's mind. Addendum: nor his body.
arisAlexis 4y ago
There are top level scientistis warning of this apocalypse. Sam Altman which was the CEO of the very site you are posting and the CEO of OpenAI has warned starkly that there is a probability that humanity will go extinct from AI in his latest interview. Have you seen it? You think he is somehow bluffing or is he part of the same cult?
- deleted 4y ago
  [deleted]
- fidgewidge 4y ago
  Sam Altman isn't a scientist but even if he was, so what, scientists have a reputation for constantly claiming the world will end unless they get more grant funding. Fear is a great motivator and makes people feel important.
nathan_young 4y ago
Ad what % is it crazy shit? I'm at about 7% of human disempowerment/exctinction, so it probably won't happen. Am I crazy?
- simple-thoughts 4y ago
  You’re vastly overestimating the risks but you’re also not crazy - humans naturally respect authority figures and follow their beliefs which is in fact good for a healthy life.
corobo 4y ago
I'm sure a lot of people are basing their belief on AI on sci-fi interpretation. If you're one of the folks doing this at least acknowledge the story would have been quite boring if it was 800 pages of "everything went well, the AI turned out really great, very helpful. Would use again" If you're not blindly afraid of zombies you shouldn't be blindly afraid of AI. Give it thought. If you're still scared after dismissing sci-fi bias, cool, I'm all ears.
- AgentME 4y ago
  The AI safety community has many people who were previously extremely optimistic about the sheer world-changing potential of AI, but then realized that getting AI right might be harder than getting it wrong, and that figuring out how to get it right must be a priority before a superintelligent AI is made. It's not people who just saw Terminator and made up their mind.
  - corobo 4y ago
    Can you back up those claims? I don't believe all of the nuance is contained in your comment. I'm looking into MIRI, if that's not "the AI safety community" please do correct me. I've become very aware ever since ChatGPT started saying inaccurate things confidently that humans have a much worse hit rate. Seriously, keep an eye out for it, you'll see it everywhere. At least ChatGPT will double check if you ask if it's sure. People tend to just get annoyed when you don't blindly trust their "research" haha.
    - AgentME 4y ago
      Eliezer Yudkowsky and the LessWrong forum popularized AI safety/alignment ideas. (The Effective Altruism community was originally mostly populated by people from LessWrong.) I think this article is a little awkwardly infatuated with LessWrong, I say even as a fan, but it does fit this discussion conveniently well about where they're coming at the subject from: https://unherd.com/2020/12/how-rational-have-you-been-this-year/ https://unherd.com/2020/12/how-rational-have-you-been-this-y... While MIRI is prominently connected to Yudkowsky, I wouldn't treat them as defining the AI alignment community. There are many people not involved with it who make substantive posts and discussions on LessWrong and the Alignment Foundations forum. There are other organizations too. OpenAI considers alignment important and has researchers concerned with it, though Yudkowsky argues the company doesn't do enough to prioritize it relative to the AI progress they make. Anthropic is an AI company prioritizing AI safety through interpretability research.
gwd 4y ago
> AI safety is an apocalypse cult Just because the parts of the community in the Effective Altruism / AI Safety have developed aspects of a cult doesn't mean that their arguments are wrong. It just means that they're humans doing human things. It sounds like their main mistake was to not realize that the cult-like behavior of other cults was sociological, due to the fact that they contained humans, as opposed to from their belief systems; and thus didn't take any steps to try to prevent their own movement from developing cult-like aspects. That said, if you have a movement of thousands (?) of people all trying to figure out how to make AI safe for over a decade, and you still don't feel like you're any closer to that goal at the end of it, then you should probably step back and ask yourself what you're doing wrong.
smk_ 4y ago
John von Neumann, who contributed to the Manhattan project, published an essay called “Can we Survive Technology?” on the topic: https://geosci.uchicago.edu/~kite/doc/von_Neumann_1955.pdf https://geosci.uchicago.edu/~kite/doc/von_Neumann_1955.pdf in 1955.
afiori 4y ago
This is a very inductivist turkey approach to safety.
weatherlight 4y ago
Extinction is the rule, Survivability is the exception.

JamesBarney 4y ago

There have been people for a long time who think the world is going to end. We can use your argument to dismiss anyone who fears any existential risk like asteroids, nuclear war, climate change, AI, etc... And since the world exists every group will be wrong except one. I think the risk of human extermination is much lower than many in this group believe because I think AI is much more compute bound than algorithm bound. So I think a hard takeoff is very unlikely. But while the median case for climate change is probably worse than the median case for AI. The existential risks for AI seem much higher. And it's a concern worth taking seriously.

esperent 4y ago
The issue is that the effective altruism crowd are focused exclusively on AI safety, at least in what they publish online. If they talked about all possible threats to humanity and how to solve them, from AI, to climate change, to nuclear bombs, to asteroids, to disease, they would be an interesting and important group. They don't. They hardly talk about anything else except AI. From the outside it looks a lot like an obsession. Personally I think they treat AI as an existential threat above all others because they consider themselves to be the pinnacle of intelligence in the universe and AI is showing that human intelligence might not be so special after all. So it's an existential threat to their sense of self. Other existential threats merely threaten their lives and are therefore not as important. This would be fine, if acknowledged with a degree of self awareness. But it's not, instead it's wrapped up in so many layers of convoluted logic that most people, insiders and outsiders both, can't see the real reason for this obsession.
- rhn_mk1 4y ago
  You might want to get familiar with what effective altruism actually focuses on. https://www.effectivealtruism.org/articles/introduction-to-effective-altruism#what-are-some-examples-of-effective-altruism-in-practice https://www.effectivealtruism.org/articles/introduction-to-e... https://www.lesswrong.com/posts/JmmA2Mf5GrY9D6nQD/four-focus-areas-of-effective-altruism https://www.lesswrong.com/posts/JmmA2Mf5GrY9D6nQD/four-focus... https://www.givingwhatwecan.org/best-charities-to-donate-to-2023#donate-to-expertmanaged-funds https://www.givingwhatwecan.org/best-charities-to-donate-to-...
  - deleted 4y ago
    [deleted]
- mitthrowaway2 4y ago
  I think they're worried about it because they honestly do care about existential risk. If you can come up with some more likely existential threats than AI, I'm sure they'll worry about that too.
- yanderekko 4y ago
  >The issue is that the effective altruism crowd are focused exclusively on AI safety. That's really not true. There's a huge overlap in these groups, but the top EA charities are not shoveling all their money at AI alignment orgs... even if one can argue that they should.
  - whatshisface 4y ago
    Arguing that AI safety is needed only demonstrates that it's altruistic, but you have a lot farther to go if you want to say funding research in this decade can be effective.
- Natsu 4y ago
  > The issue is that the effective altruism crowd are focused exclusively on AI safety. How many EA people do you know? That seems to get the most clicks, but I usually associate them with things like mosquito bed nets to help prevent malaria, because that's a far more common topic in my experience. I'm sure you can find particular people who are all about the AI risk stuff that seems pretty wild to me, too. But when you say "focused exclusively" I feel like you may have mostly seen news articles about wild AI stuff and not the average EA enthusiast who spends their time talking about picking efficient charities.
zarzavat 4y ago
> AI is much more compute bound than algorithm bound Any reason to believe this? The human brain runs on 20 watts (!). Obviously biological hardware is not directly comparable to silicon, perhaps silicon is intrinsically less efficient by some factor such that in order to obtain 20W of human brain compute we need to expend K times more in silicon. But is that factor really so large that we are compute-bound? How big would K have to be for us to be compute-bound anyway? If you could have Einstein for 200kW (i.e. silicon is intrinsically 1000x less efficient than brains) would that be compute-bound? It seems much more likely that we are algorithm-bound and simply not using our available compute efficiently.
- pas 4y ago
  So there are many failure modes possible here. Yudkowsky's hard take off mentions "solving protein folding", nanomachines, persuading everyone to let them connect to the internet, infinite adaptability, etc. I think those are unrealistic, but obviously they still have a non-zero chance. More probably bad possibilities are AI-Hitler (monomaniacal populist absolute leader) or AI-Stalin (manipulative, smart, absolutely paranoid, rising through the ranks) ... so something that's human-like enough to be able to connect with humans, to manipulate them, but at the same time less affected by the psychological shortcomings. (Ie. such an AI could spend enough time to cross-interrogate every underling, constantly watch them, etc.) And yes, a very efficient immortal dictator is very bad news, but still bound by human-like limits. And the big infinite dollar question is could this hypothetical AI improve on itself by transcending human limits? Let's say by directly writing programs that it has conscious control over? Can it truly "watch" a 1000 video streams in real-time? Can it increase the number of its input-and-output channels while maintaining its human-like efficiency? Because it's very different to run a fast neural network that spits out a myriad labels for every frame of a video stream (YOLO does this already, but it's not 20W!) and to integrate those labels into actions based on a constantly evolving strategy. Sure maybe the hypothetical AI will simply run a lot of AlphaZero-like hybrid tree-search estimator-evaluator things ... Anyway, what I'm trying to say is that our 20W efficiency comes with getting tried very fast, and using "fast mode" thinking for everything. (Slow mode is the exception, and using it is so rare that we basically pop open a glass of champagne every time.)
paulcole 4y ago
> The existential risks for AI seem much higher What’s the upper bound for climate change’s existential risks? The end of human existence and society as we know it except for a relatively small number of survivors living in a world those of us alive today can barely imagine?
- fidgewidge 4y ago
  There is no realistic scenario provided for by current scientific knowledge that would lead to that outcome. Waterworld was a film, not a textbook.
  - mitthrowaway2 4y ago
    The scenario described in this article comes the closest, detailing a mechanism that may have been responsible for the end-Permian mass extinction, wherein warming oceans become anoxic and begin to release huge quantities of hydrogen sulfide gas. The gas is directly lethal to most life in the oceans and on land, and destroys the ozone layer as a bonus. It's hard for me to imagine any way in which an agricultural human civilization could survive that scenario. https://www.scientificamerican.com/article/impact-from-the-deep/ https://www.scientificamerican.com/article/impact-from-the-d...
    - fidgewidge 4y ago
      By "climate change" people mean the sort of change that will be realistically caused by humans within the next few hundred years, not the sort of extinction caused by massive volcano activity that takes 10,000 years and happens once every 100 million years. Obviously if the climate changes drastically enough fast enough then everyone dies but nobody is suggesting that's the case for preset day human activity.
      - mitthrowaway2 4y ago
        In terms of timeline, the key sentence would be "The so-called thermal extinction at the end of the Paleocene began when atmospheric CO2 was just under 1,000 parts per million (ppm)". The end-Permian was likely around 2500 ppm[1]. It doesn't really matter whether that carbon comes from supervolcanoes or from human emissions. When the article was written, CO2 concentration was 385 ppm. Today it's 421 ppm. If CO2 concentrations were to rise linearly, the timeline for reaching 1,000 ppm would be 250 years. Achieving that would require emissions to stabilize at ~2014 levels. If emissions keep rising, then perhaps it's closer to 150 years. If strong feedbacks kick in, like methane from melting permafrost, then maybe 100 years or less, or maybe we'll reach that happy 2500 ppm mark and see what a real mass-extinction looks like. Of course the ocean has a big thermal mass, so it will probably take quite some time to heat up enough to trigger the disaster even after we reach that level of atmospheric carbon. Hopefully everything will be fine? [1] https://www.nature.com/articles/s41467-021-22298-7 https://www.nature.com/articles/s41467-021-22298-7
freejazz 4y ago
Numbers used to justify ignoring common sense
scotty79 4y ago
> And since the world exists every group will be wrong except one. Except at most one.
ElfinTrousers 4y ago
Asteroids, nuclear war, climate change, and AI. As the old song says "One of these things is not like the other". We know that asteroids exist and many large ones are uncomfortably close to our home. We know that nuclear weapons exist and have seen them used. We know that the climate is getting hotter (*). AI is...well, people losing their shit over ChatGPT (**) aside, AI is not going to be real enough to worry about for a few more decades at least. (*) Anyone who's about to regurgitate some fossil-fuel industry talking points in response, just save your breath. (**) Above note goes double for this.
- JohnPrine 4y ago
  Would you rather we wait until AIs are actually posing a threat for us to study ways to align them with human values? Tons of money already goes into fighting climate change and basically everyone on earth is aware of the threat it poses. The AI safety field is only about a decade old and is relatively unknown. Of course it makes sense to raise awareness there
  - ElfinTrousers 4y ago
    > Would you rather we wait until AIs are actually posing a threat for us to study ways to align them with human values? No, I would rather we wait until we are close enough to AI that we are talking about something concrete, rather than making wild speculations.
    - JohnPrine 4y ago
      Interesting. In your opinion, how should we decide when we're close enough that we're talking about something concrete?
      - ElfinTrousers 4y ago
        Please see second asterisked point above.

voltaireodactyl 4y ago

On the other hand, we and we alone live in a time where a grapefruit-sized weapon could obliterate a reasonably sized town, AND when such weapons are increasingly in reach of non-state entities. I would imagine the chance of apocalypse rises as fewer people in agreement would be required to enact it.

deleted 4y ago

[deleted]

rvz 4y ago

I don't think there is anything safe about ChatGPT regurgitating and hallucinating false information about someone's business based on real legal proceedings that it had been trained on. [0] As long as it cannot also explain its own decisions transparently [1], then there there will always be a need for AI safety. [0] https://news.ycombinator.com/item?id=35063005 https://news.ycombinator.com/item?id=35063005 [1] https://news.ycombinator.com/item?id=35060285 https://news.ycombinator.com/item?id=35060285

thorum 4y ago

While you make a reasonable point in terms of human psychology, when you create an AI, you are leaving the realm of human psychology and dealing with something new that doesn’t have to play by the same rules. I’d argue that’s a big difference. We’re creating a fundamentally alien intelligence from scratch and basically just crossing our fingers and hoping to hell it cares about keeping us alive.

fshbbdssbbgdd 4y ago

Personally, I find the AI safety arguments sound pretty far-fetched, but I can’t come up with solid arguments against them. Also, when I look for a good ”debunking”, I don’t find anything. Mostly all I find are mockery, ad-hominem and the like. Frankly, the people who argue climate change isn’t real do a better job than the people who argue AI risk isn’t real, and they’re wrong!

j_crick 4y ago
> but I can’t come up with solid arguments against them. Let chaos take the world! /s
Vecr 4y ago
Yeah, I can't really argue against the whole idea. My "P(doom)"(language from the article), at least as related to agentic AI, is pretty low. About 7 or 8 percent before I die, and I'm not really old. There's probably not a lot I could do about it anyway, given that most AI jobs are alleged to just make it worse. That said, I can't really go around telling people with different numbers they should be working on antiviral research or physical manufacturing, just because I (personally, without their massive and cohesive "evidence" (i.e. gamed out scenarios) bases) think that's more important.
- vcxy 4y ago
  "Roll a D20 and if you get 1 humanity as you know it will be destroyed in your lifetime" seems bad. Why do you consider 7 or 8 percent low?
  - Vecr 4y ago
    I'm not one of those "all value in the universe" people, P(doom) for me does not have to be that bad. I would actually consider the inability to manufacture or unwillingness to manufacture certain things at scale in the US to count. Generally, my point is that agentic AI is not something I can really do much about, and other things have a combined P(doom) probability higher than it. The unwillingness in certain groups to use what they consider to be toxic or "forever" chemicals in the manufacture of physical products at scale that can make them world a better and less fragile place is something that could be worked in instead.

yanderekko 4y ago

"Apocalypse cults are always wrong, AI safety is an apocalypse cult, therefore it is (probably) wrong" is an argument which really only betrays the ignorance of the speaker regarding the specifics of the AI doomer argument. Why are apocalypse cults always wrong? Do the same specific reasons apply to the doomers here?

geraldhh 4y ago
> Why are apocalypse cults always wrong? because we are still here
- yanderekko 4y ago
  That's the consequence of them being wrong, not the cause..
  - geraldhh 4y ago
    denying the existence of circular dependencies does not invalidate them
- roenxi 4y ago
  If there was an apocalypse cult in ancient Pompeii predicting the mountain would explode and bury everyone with ash, they would not be here. And there easily could have been. This is not an argument, it is a tautology powered by the anthropic principle.
  - geraldhh 4y ago
    that example is a local disaster, not an apocalypse. but yes, there most certainly were people predicting it and they probably didn't get a chance to enjoy being right.
simple-thoughts 4y ago
Humans are attracted to certain types of mythological stories - and there’s nothing wrong with that. It becomes a problem when the stories and reality are conflated, like is happening with AI. The idea that apocalypse cults are always wrong comes from two source: (1) empirical evidence, every time there’s been one it’s wrong and (2) understanding human sociology, humans are susceptible to making certain types of thought errors as groups and apocalypse cults such as the AI one demonstrate those errors everywhere. So yeah an apocalypse cult could be right, but it would necessarily be for the wrong reasons. Just like a broken clock that matches the current time 2 minutes of the full day.

precompute 4y ago

IMO AI Safety is to AI as HR is to Employees. You can't just learn it out of a book and even then it's a pretty bad experience for everyone involved. The world is not a corporation and the existence of "AI Safety" just guarantees that no "mere mortals" will ever get to touch AI. Waiting for the day our Overlords make Graphic Card 2.0 and refuse to sell any to the plebs 'cause the overlords "know better"

kelnage 4y ago

Absolutely agree. My biggest problem with AI safety is that, simply, the problem they envisage doesn't exist yet (generally, at a minimum, relying on the existence of "AGI"). Hence discussions about it have to make a huge amount of assumptions about a whole range of aspects of what the AI threat will be - what the AI will be capable of, what it's impact will be - before getting on to what possible solutions might be relevant to preventing it. But given the first two are so undefined, the later is pure speculation - one that is difficult to criticise directly, because any specific critisms can usually be easy deflected by adjusting any of the above assumptions without making a substantial change to the "inevitable" conclusion. That's why it feels like an apocalypse cult to me - it's a conclusion, that has little strong evidence today, stacked on top of a constantly shifting set of assumptions, allowing adherents to avoid backing their arguments with evidence.

shapefrog 4y ago
For the freseable future the real danger I see from AI is inferior basic statistical models being jammed into production pretending to be some sort of all seeing all knowing AGI - a class of product we are not really that close to.

hiAndrewQuinn 4y ago

I don't have any solid arguments either, but I just get the feeling that after the first very public tragedy, like a killbot rampaging in New York City, we'll see a very, very sudden tilt in the public eye towards never wanting to touch this technology again. People are terrified of killer robots. One of the likely outcomes like that is even stronger pivoting towards walled gardens and end to end stuff than we previously had on the Internet. Can't train a model on someone else's encrypted database without decrypting it first.

CatWChainsaw 4y ago
Thou shalt not make a machine in the likeness of a man's mind. Addendum: nor his body.

arisAlexis 4y ago

There are top level scientistis warning of this apocalypse. Sam Altman which was the CEO of the very site you are posting and the CEO of OpenAI has warned starkly that there is a probability that humanity will go extinct from AI in his latest interview. Have you seen it? You think he is somehow bluffing or is he part of the same cult?

deleted 4y ago
[deleted]
fidgewidge 4y ago
Sam Altman isn't a scientist but even if he was, so what, scientists have a reputation for constantly claiming the world will end unless they get more grant funding. Fear is a great motivator and makes people feel important.

nathan_young 4y ago

Ad what % is it crazy shit? I'm at about 7% of human disempowerment/exctinction, so it probably won't happen. Am I crazy?

simple-thoughts 4y ago
You’re vastly overestimating the risks but you’re also not crazy - humans naturally respect authority figures and follow their beliefs which is in fact good for a healthy life.

corobo 4y ago

I'm sure a lot of people are basing their belief on AI on sci-fi interpretation. If you're one of the folks doing this at least acknowledge the story would have been quite boring if it was 800 pages of "everything went well, the AI turned out really great, very helpful. Would use again" If you're not blindly afraid of zombies you shouldn't be blindly afraid of AI. Give it thought. If you're still scared after dismissing sci-fi bias, cool, I'm all ears.

AgentME 4y ago
The AI safety community has many people who were previously extremely optimistic about the sheer world-changing potential of AI, but then realized that getting AI right might be harder than getting it wrong, and that figuring out how to get it right must be a priority before a superintelligent AI is made. It's not people who just saw Terminator and made up their mind.
- corobo 4y ago
  Can you back up those claims? I don't believe all of the nuance is contained in your comment. I'm looking into MIRI, if that's not "the AI safety community" please do correct me. I've become very aware ever since ChatGPT started saying inaccurate things confidently that humans have a much worse hit rate. Seriously, keep an eye out for it, you'll see it everywhere. At least ChatGPT will double check if you ask if it's sure. People tend to just get annoyed when you don't blindly trust their "research" haha.
  - AgentME 4y ago
    Eliezer Yudkowsky and the LessWrong forum popularized AI safety/alignment ideas. (The Effective Altruism community was originally mostly populated by people from LessWrong.) I think this article is a little awkwardly infatuated with LessWrong, I say even as a fan, but it does fit this discussion conveniently well about where they're coming at the subject from: https://unherd.com/2020/12/how-rational-have-you-been-this-year/ https://unherd.com/2020/12/how-rational-have-you-been-this-y... While MIRI is prominently connected to Yudkowsky, I wouldn't treat them as defining the AI alignment community. There are many people not involved with it who make substantive posts and discussions on LessWrong and the Alignment Foundations forum. There are other organizations too. OpenAI considers alignment important and has researchers concerned with it, though Yudkowsky argues the company doesn't do enough to prioritize it relative to the AI progress they make. Anthropic is an AI company prioritizing AI safety through interpretability research.

gwd 4y ago

> AI safety is an apocalypse cult Just because the parts of the community in the Effective Altruism / AI Safety have developed aspects of a cult doesn't mean that their arguments are wrong. It just means that they're humans doing human things. It sounds like their main mistake was to not realize that the cult-like behavior of other cults was sociological, due to the fact that they contained humans, as opposed to from their belief systems; and thus didn't take any steps to try to prevent their own movement from developing cult-like aspects. That said, if you have a movement of thousands (?) of people all trying to figure out how to make AI safe for over a decade, and you still don't feel like you're any closer to that goal at the end of it, then you should probably step back and ask yourself what you're doing wrong.

smk_ 4y ago

John von Neumann, who contributed to the Manhattan project, published an essay called “Can we Survive Technology?” on the topic: https://geosci.uchicago.edu/~kite/doc/von_Neumann_1955.pdf https://geosci.uchicago.edu/~kite/doc/von_Neumann_1955.pdf in 1955.

afiori 4y ago

This is a very inductivist turkey approach to safety.

weatherlight 4y ago

Extinction is the rule, Survivability is the exception.