6 ms·

There's this weird disconnect in tech circles, where everyone is deathly afraid of AGI, but totally asleep on the very real possibility of thermonuclear war bre

There's this weird disconnect in tech circles, where everyone is deathly afraid of AGI, but totally asleep on the very real possibility of thermonuclear war breaking out in Europe or Asia over the next 10 years. There's already credible evidence that we came perilously close to the use of tactical nuclear weapons in Ukraine which likely would've spiraled out of control. AGI might happen, but the threat of nuclear war keeps me up at night.

runjake 1y ago
100%. Except the things that kill most of us[1] keeps me up at night. 1. https://ourworldindata.org/does-the-news-reflect-what-we-die-from https://ourworldindata.org/does-the-news-reflect-what-we-die...
- masfuerte 1y ago
  I'm more concerned with the ways that other people can kill me than with the ways they kill themselves.
myrmidon 1y ago
I personally think that AI is a realistic extinction threat for our whole species within this century, and a full nuclear war never was (and probably never will be). Neither is climate change. Collapse of our current civilizations? Sure. Extinction? No. And I honestly see stronger incentives on a road towards us being outcompeted by AI, then on our leaders starting a nuclear war.
- lm28469 1y ago
  Personally I don't bieleve in a collapse nor extinction, just a slow spiral into more and more enshittification. You'll have to talk to an "ai" doctor because real doctors will treat people with money, you'll have to face an "ai" administration because the real administration will work for people with money, you'll have to be a flesh and blood robot to an "ai" telling you what to do (already the case for Amazon warehouse workers, food delivery people, &c.), some "ai" will determine if you qualify for X or Y benefits, X or Y treatment, X or Y job. Basically everything wrong with today's productivism, but 100 times worse and powered by a shitty ai that's very far from agi.
- johnnyanmac 1y ago
  Total extinction of any dominant species is really hard. Very few post-apocalyptic settings suggest a full extinction and usually show some thousands of survivors struggling with the new norm. Humans in particular are very adaptable so thorough killing all 8 billion of us would be difficult no matter the scenario. I think only the Sun can do that and that's assuming we fail to find an exit strategy 5 billion years in (we're less than a thousandth of a percent into humanity if we measure on that scale). As such, I'd say "extinction" is more of a colloquial use of "Massive point in history that kills off billions in short order".
- audunw 1y ago
  Why? There’s nothing in the current process of developing AI that would lead to a AI that would act against humanity of its own choosing. The development process is hyper-optimised to make AIs that do exactly what humans tell it to do. Sure, an LLM AI can role-play as an evil super AI out to kill humans. But it can just as easily role-play as one that defends humanity. So that tells us nothing about what will happen. We could just as well think that exploding the first nuclear bomb would ignite the atmosphere and kill all of humanity. There was nothing from physics that indicated it was possible but some still thought about it. IMO that kind of thinking is pointless. Same with thinking LHC would create a black hole. As far as I can tell the fear of super intelligent AI will kill humans all boils down to something utterly magical happens and then somehow super intelligent evil AI appears.
  - myrmidon 1y ago
    > Why? There’s nothing in the current process of developing AI that would lead to a AI that would act against humanity of its own choosing. If we had certainty that our designs were categorically incapable of acting in their own interest then I would agree with you, but we absolutely don't, and I'd argue that we don't even have that certainty for current-generation LLMs. Long term, we're fundamentally competing with AI for ressources. > We could just as well think that exploding the first nuclear bomb would ignite the atmosphere and kill all of humanity. There was nothing from physics that indicated it was possible but some still thought about it. This is incorrect. Atmospheric ignition was a concern by nuclear physicists based on physics, but dismissed as unlikely after doing the math (see https://en.wikipedia.org/wiki/Effects_of_nuclear_explosions#Atmospheric_ignition https://en.wikipedia.org/wiki/Effects_of_nuclear_explosions#...). > As far as I can tell the fear of super intelligent AI will kill humans all boils down to something utterly magical happens and then somehow super intelligent evil AI appears. Not necessary at all. AI acting in its own interests and competing "honestly" with humans is already enough. This is exactly how we outcompeted every other animal on the planet after all.
squidbeak 1y ago
Other doomsday risks aren't any reason to turn our heads away from this one. AI's much more likely to end up taking an apocalyptic form if we sleep on it.
- teucris 1y ago
  But this isn’t a suggestion to turn away from AI threats - it’s a matter of prioritization. There are more imminent threats that we know can turn apocalyptic that swaths of people in power are completely ignoring and instead fretting over AI.
- woeirua 1y ago
  We should worry more about doomsday risks that are concrete and present today. Despite the prognostications of the uber wealthy, the emergence of AGI is not guaranteed. It likely will happen at some point, but is that tomorrow or 200 years in the future? We can’t know for sure.
- audunw 1y ago
  Why do people say it’s likely? There’s nothing in science that indicates that it’s probable. The closest field of science we can use to predict the behaviour of intelligent agents is evolution. The behaviour of animals is highly dictated by the evolutionary pressure they experience in their development. Animals kill other animals when they need to compete for resources for survival. Now think about the evolutionary pressure for AIs. Where’s the pressure towards making AIs act on their own behalf to compete with humans? Let’s say there’s somehow something magical that pushes AIs towards acting in their own self interest at the expense of humans. Why do we believe they will go from 0% to 100% efficient and successful at this in the span of what? Months? It seems more likely that there would be years of failed attempts at breaking out, before a successful attempt is even remotely likely. This would just further increase the evolutionary pressure humans exerts on the AIs to stay in line with our expectations. Attempting to eliminate your creators is fundamentally a pretty stupid action. It seems likely that we will see thousands of attempts by incompetent AIs before we see one by a truly superintelligent one.
marssaxman 1y ago
Or, you know, the bit where we've now-irrevocably committed ourselves to destabilizing the global climate system whose relative predictability has been the foundation of our entire civilization. That's going to be a ride.
- bee_rider 1y ago
  We just need to convince people that the market is an artificial superintelligence. Or maybe… subintelligence.
  - marssaxman 1y ago
    Years ago a friend of mine observed that we don't need to wonder what it would look like if artificial entities were to gain power and take over our civilization, because it already happened: we call them "corporations".
nradov 1y ago
Well it's not everyone. I guess I am in "tech circles" and have zero fear of AGI. Everyone who is (or claims to be) "deathly afraid" is either ignorant or unserious or a grifter. Their arguments are essentially a form of secular religion lacking any firm scientific basis. These are not people worth listening to.
zeroonetwothree 1y ago
Everyone thinks their own field is the most important and deserving of attention and funding. Big surprise.
jjtheblunt 1y ago
> credible evidence that we came perilously close to the use of tactical nuclear weapons in Ukraine I've not seen that. can you link to it?
j2kun 1y ago
> everyone is deathly afraid of AGI I think this is a vast overstatement. A small group of influential people are deathly afraid of AGI, or at least using that as a pretext to raise funding. But I agree that there are so many more things we should be deathly afraid of. Climate change tops my personal list as the biggest existential threat to humanity.
- johnnyanmac 1y ago
  I sure wish my conspiracy theories could lead to me running billion dollar projects to defend against my shadow demons. Instead I just get ratio'd by the internet and get awkward silences at family gatherings. I think the sad part is that most people in power aren't planning to be around in 10 years, so they don't care about any long term issues that are cropping up. leave it to their grandchildren to burn with the world.
nonethewiser 1y ago
>There's already credible evidence that we came perilously close to the use of tactical nuclear weapons in Ukraine which likely would've spiraled out of control. I do agree nukes are a far more realistic threat. So this is kind of an aside and doesn't really undermine your point. But I actually think we widely misunderstand the dynamic of using nuclear weapons. Nukes haven't been used for a long time and everyone kind of assumes using them will inevitably lead to escalation which spirals into total destruction. But how would Russia using a tactical nuke in Ukraine spiral out of control? It actually seems very likely that it would not be met in kind. Which is absolutely terrifying in it's own right. A sort of normalization of nuclear weapons.
- aradox66 1y ago
  It's not an assumption, it's an extremely developed international field of tactical and strategic study that leads to these conclusions
  - nonethewiser 1y ago
    Extremely developed thought experiment maybe. Only 2 nuclear weapons have ever been dropped. Which is why I say it's a massive assumption. You tell me. How does this escalate into a total destruction scenario? Russia uses a small nuke on a military target in the middle of nowhere Ukraine. ___________________________. Everyone is firing nukes at eachother. Fill in the blank. We are not talking about the scenario where Russia fire a nuke at Washington, Colorado, CA, Montana, forward deployments, etc. and the US responds in kind while nukes are en-route.
    - johnnyanmac 1y ago
      >How does this escalate into a total destruction scenario? My favorite historical documentary: https://www.youtube.com/watch?v=Pk-kbjw0Y8U https://www.youtube.com/watch?v=Pk-kbjw0Y8U (my new favorite part is America realizing "fuck, we're dumasses" far too late into the warfare they started). That is to say: you're assuming a lot of good faith in a time of unrest with several leaders looking for any excuse enact martial law. For all we know, the blank is "Trump overreacts and authorizes a nuclear strike on Los Angeles"(note the word "authorizes". Despite the media, the president cannot unilaterally fire a nuclear warhead). That bizarre threat alone might escalate completely unrelated events and boom. Chaos.
Stevvo 1y ago
There is no evidence that use of tactical nuclear weapons in Ukraine would spiral out of control. I like to think that the US/UK/France would stay out of it, if only because the leaders value their own lives if not those of others.
proto-n 1y ago
Well, one of these is something that most reaonable people work on avoiding, while the other is something that a huge capitalist industrial machine is working to achieve like their existence depends on it.
hearsathought 1y ago
> There's already credible evidence that we came perilously close to the use of tactical nuclear weapons in Ukraine which likely would've spiraled out of control. Spiral out of control in what way? Wouldn't it have ended the war immediately.
shadowpho 1y ago
>There's already credible evidence that we came perilously close to the use of tactical nuclear weapons in Ukraine which likely would've spiraled out of control Do you want to share any of this credible evidence?

runjake 1y ago

100%. Except the things that kill most of us[1] keeps me up at night. 1. https://ourworldindata.org/does-the-news-reflect-what-we-die-from https://ourworldindata.org/does-the-news-reflect-what-we-die...

masfuerte 1y ago
I'm more concerned with the ways that other people can kill me than with the ways they kill themselves.

myrmidon 1y ago

I personally think that AI is a realistic extinction threat for our whole species within this century, and a full nuclear war never was (and probably never will be). Neither is climate change. Collapse of our current civilizations? Sure. Extinction? No. And I honestly see stronger incentives on a road towards us being outcompeted by AI, then on our leaders starting a nuclear war.

lm28469 1y ago
Personally I don't bieleve in a collapse nor extinction, just a slow spiral into more and more enshittification. You'll have to talk to an "ai" doctor because real doctors will treat people with money, you'll have to face an "ai" administration because the real administration will work for people with money, you'll have to be a flesh and blood robot to an "ai" telling you what to do (already the case for Amazon warehouse workers, food delivery people, &c.), some "ai" will determine if you qualify for X or Y benefits, X or Y treatment, X or Y job. Basically everything wrong with today's productivism, but 100 times worse and powered by a shitty ai that's very far from agi.
johnnyanmac 1y ago
Total extinction of any dominant species is really hard. Very few post-apocalyptic settings suggest a full extinction and usually show some thousands of survivors struggling with the new norm. Humans in particular are very adaptable so thorough killing all 8 billion of us would be difficult no matter the scenario. I think only the Sun can do that and that's assuming we fail to find an exit strategy 5 billion years in (we're less than a thousandth of a percent into humanity if we measure on that scale). As such, I'd say "extinction" is more of a colloquial use of "Massive point in history that kills off billions in short order".
audunw 1y ago
Why? There’s nothing in the current process of developing AI that would lead to a AI that would act against humanity of its own choosing. The development process is hyper-optimised to make AIs that do exactly what humans tell it to do. Sure, an LLM AI can role-play as an evil super AI out to kill humans. But it can just as easily role-play as one that defends humanity. So that tells us nothing about what will happen. We could just as well think that exploding the first nuclear bomb would ignite the atmosphere and kill all of humanity. There was nothing from physics that indicated it was possible but some still thought about it. IMO that kind of thinking is pointless. Same with thinking LHC would create a black hole. As far as I can tell the fear of super intelligent AI will kill humans all boils down to something utterly magical happens and then somehow super intelligent evil AI appears.
- myrmidon 1y ago
  > Why? There’s nothing in the current process of developing AI that would lead to a AI that would act against humanity of its own choosing. If we had certainty that our designs were categorically incapable of acting in their own interest then I would agree with you, but we absolutely don't, and I'd argue that we don't even have that certainty for current-generation LLMs. Long term, we're fundamentally competing with AI for ressources. > We could just as well think that exploding the first nuclear bomb would ignite the atmosphere and kill all of humanity. There was nothing from physics that indicated it was possible but some still thought about it. This is incorrect. Atmospheric ignition was a concern by nuclear physicists based on physics, but dismissed as unlikely after doing the math (see https://en.wikipedia.org/wiki/Effects_of_nuclear_explosions#Atmospheric_ignition https://en.wikipedia.org/wiki/Effects_of_nuclear_explosions#...). > As far as I can tell the fear of super intelligent AI will kill humans all boils down to something utterly magical happens and then somehow super intelligent evil AI appears. Not necessary at all. AI acting in its own interests and competing "honestly" with humans is already enough. This is exactly how we outcompeted every other animal on the planet after all.

squidbeak 1y ago

Other doomsday risks aren't any reason to turn our heads away from this one. AI's much more likely to end up taking an apocalyptic form if we sleep on it.

teucris 1y ago
But this isn’t a suggestion to turn away from AI threats - it’s a matter of prioritization. There are more imminent threats that we know can turn apocalyptic that swaths of people in power are completely ignoring and instead fretting over AI.
woeirua 1y ago
We should worry more about doomsday risks that are concrete and present today. Despite the prognostications of the uber wealthy, the emergence of AGI is not guaranteed. It likely will happen at some point, but is that tomorrow or 200 years in the future? We can’t know for sure.
audunw 1y ago
Why do people say it’s likely? There’s nothing in science that indicates that it’s probable. The closest field of science we can use to predict the behaviour of intelligent agents is evolution. The behaviour of animals is highly dictated by the evolutionary pressure they experience in their development. Animals kill other animals when they need to compete for resources for survival. Now think about the evolutionary pressure for AIs. Where’s the pressure towards making AIs act on their own behalf to compete with humans? Let’s say there’s somehow something magical that pushes AIs towards acting in their own self interest at the expense of humans. Why do we believe they will go from 0% to 100% efficient and successful at this in the span of what? Months? It seems more likely that there would be years of failed attempts at breaking out, before a successful attempt is even remotely likely. This would just further increase the evolutionary pressure humans exerts on the AIs to stay in line with our expectations. Attempting to eliminate your creators is fundamentally a pretty stupid action. It seems likely that we will see thousands of attempts by incompetent AIs before we see one by a truly superintelligent one.

marssaxman 1y ago

Or, you know, the bit where we've now-irrevocably committed ourselves to destabilizing the global climate system whose relative predictability has been the foundation of our entire civilization. That's going to be a ride.

bee_rider 1y ago
We just need to convince people that the market is an artificial superintelligence. Or maybe… subintelligence.
- marssaxman 1y ago
  Years ago a friend of mine observed that we don't need to wonder what it would look like if artificial entities were to gain power and take over our civilization, because it already happened: we call them "corporations".

nradov 1y ago

Well it's not everyone. I guess I am in "tech circles" and have zero fear of AGI. Everyone who is (or claims to be) "deathly afraid" is either ignorant or unserious or a grifter. Their arguments are essentially a form of secular religion lacking any firm scientific basis. These are not people worth listening to.

zeroonetwothree 1y ago

Everyone thinks their own field is the most important and deserving of attention and funding. Big surprise.

jjtheblunt 1y ago

> credible evidence that we came perilously close to the use of tactical nuclear weapons in Ukraine I've not seen that. can you link to it?

j2kun 1y ago

> everyone is deathly afraid of AGI I think this is a vast overstatement. A small group of influential people are deathly afraid of AGI, or at least using that as a pretext to raise funding. But I agree that there are so many more things we should be deathly afraid of. Climate change tops my personal list as the biggest existential threat to humanity.

johnnyanmac 1y ago
I sure wish my conspiracy theories could lead to me running billion dollar projects to defend against my shadow demons. Instead I just get ratio'd by the internet and get awkward silences at family gatherings. I think the sad part is that most people in power aren't planning to be around in 10 years, so they don't care about any long term issues that are cropping up. leave it to their grandchildren to burn with the world.

nonethewiser 1y ago

>There's already credible evidence that we came perilously close to the use of tactical nuclear weapons in Ukraine which likely would've spiraled out of control. I do agree nukes are a far more realistic threat. So this is kind of an aside and doesn't really undermine your point. But I actually think we widely misunderstand the dynamic of using nuclear weapons. Nukes haven't been used for a long time and everyone kind of assumes using them will inevitably lead to escalation which spirals into total destruction. But how would Russia using a tactical nuke in Ukraine spiral out of control? It actually seems very likely that it would not be met in kind. Which is absolutely terrifying in it's own right. A sort of normalization of nuclear weapons.

aradox66 1y ago
It's not an assumption, it's an extremely developed international field of tactical and strategic study that leads to these conclusions
- nonethewiser 1y ago
  Extremely developed thought experiment maybe. Only 2 nuclear weapons have ever been dropped. Which is why I say it's a massive assumption. You tell me. How does this escalate into a total destruction scenario? Russia uses a small nuke on a military target in the middle of nowhere Ukraine. ___________________________. Everyone is firing nukes at eachother. Fill in the blank. We are not talking about the scenario where Russia fire a nuke at Washington, Colorado, CA, Montana, forward deployments, etc. and the US responds in kind while nukes are en-route.
  - johnnyanmac 1y ago
    >How does this escalate into a total destruction scenario? My favorite historical documentary: https://www.youtube.com/watch?v=Pk-kbjw0Y8U https://www.youtube.com/watch?v=Pk-kbjw0Y8U (my new favorite part is America realizing "fuck, we're dumasses" far too late into the warfare they started). That is to say: you're assuming a lot of good faith in a time of unrest with several leaders looking for any excuse enact martial law. For all we know, the blank is "Trump overreacts and authorizes a nuclear strike on Los Angeles"(note the word "authorizes". Despite the media, the president cannot unilaterally fire a nuclear warhead). That bizarre threat alone might escalate completely unrelated events and boom. Chaos.

Stevvo 1y ago

There is no evidence that use of tactical nuclear weapons in Ukraine would spiral out of control. I like to think that the US/UK/France would stay out of it, if only because the leaders value their own lives if not those of others.

proto-n 1y ago

Well, one of these is something that most reaonable people work on avoiding, while the other is something that a huge capitalist industrial machine is working to achieve like their existence depends on it.

hearsathought 1y ago

> There's already credible evidence that we came perilously close to the use of tactical nuclear weapons in Ukraine which likely would've spiraled out of control. Spiral out of control in what way? Wouldn't it have ended the war immediately.

shadowpho 1y ago

>There's already credible evidence that we came perilously close to the use of tactical nuclear weapons in Ukraine which likely would've spiraled out of control Do you want to share any of this credible evidence?