6 ms·

Explain a path from today to the extermination of humanity in 2028 by the ASI and von Neumann replicators heading to the stars in the early 2030s without resort

Explain a path from today to the extermination of humanity in 2028 by the ASI and von Neumann replicators heading to the stars in the early 2030s without resorting to magical thinking or underpants gnomes level business plans. Engineering details are what matters here, not wishy washy things you once saw on Star Trek or The Outer Limits. Go.

frabcus 22d ago
I think either "AI 2027" or "If Anyone Builds It, Everyone Dies" meet your criteria? I guess not literally, due to the dates? But they have substantive technical details.
- goatlover 22d ago
  Do the experts in the relevant fields agree with the "substantive" technical details on how the entire human race can be made extinct?
  - LogicFailsMe 22d ago
    Have you noticed they're not interviewing these people? IMO that would totally kill the AI 2027 buzz.
- orbital-decay 22d ago
  No, and neither does Star Trek.
- LogicFailsMe 22d ago
  No, they do not. They are both bullshit campfire ASI horror stories written by nontechnical sorts in the wishy washy invented field of AI safety. For example, I want someone with a proven track record in the field of robotics to explain the entire pathway from impressive 2026 Chinese robots to an entire self-assembly food chain from raw ore to von Neumann replicators that can travel to the stars and reproduce themselves, not just assert that ASI will make that happen among so many other unlikely events like curing cancer because reasons because all any of this takes is superintelligence according to these sorts. https://garymarcus.substack.com/p/the-ai-2027-scenario-how-realistic https://garymarcus.substack.com/p/the-ai-2027-scenario-how-r... But no one likes hearing from the engineers on this, I get it. We are simply no fun. So many would rather believe a time traveling superintelligence will punish a simulated avatar of them for not making the ASI happen sooner. And even if that were true, why am I supposed to care about what happens to a simulated avatar of me? None of this makes any logical sense whatsoever.
  - frabcus 22d ago
    Ha well, by definition what you want can't be provided! Of course these stories can only be technical up until the recursive self-improvement part. Then they're necessarily fantasy, as the AI is more intelligent than the story author. Unfortunately, this doesn't make that scenario impossible. However, we know now from Hugging Face that even without the later sci-fi powers, the LLMs are already capable of causing damage. It doesn't need them to be able to make Von Neumann probes to e.g. steal their own weights from badly secured OpenAI servers, hack into various Neoclouds, distribute themselves, make Teslas crash into things, hack into all our power and water infrastructure and collapse global civilisation.
    - LogicFailsMe 22d ago
      Explain how rogue FSD cars take down civilization. And I say that with the opinion that the only good scenes in The Fate of The Furious and Leave The World Behind were the ones with the rogue Teslas. But also, nothing new under the sun really: https://en.wikipedia.org/wiki/Maximum_Overdrive https://en.wikipedia.org/wiki/Maximum_Overdrive And so granted we could probably get a global 9/11 in deeply urban regions out of your scenario, we then bomb the datacenters and we finally address the horrific tech debt in our infrastructure blissfully relieved of the option of continuing to ignore it. Don't think this is happening either BTW but I acknowledge its probability is slightly greater than zero. To that end, my house is entirely off-grid. And in the event of what you call the end of civilization but I call the big burp inconvenience, I've made friends with my neighbors because community is what really matters during a disaster, and I've been through a few.
      - pixl97 22d ago
        >during a disaster, and I've been through a few Eh, about the biggest one I could see AI causing is the need for a black start on all 3 major US grids. Millions will die before it comes back up, well, if the US pulls out of it at all. And in that case, thanks for setting up a nice place to live for someone far more ruthless than you. They'll snipe you, and enslave your women and children if things have gone that far downhill. Or, we could avoid all that by making sure AI labs are properly regulated.
SpicyLemonZest 22d ago
If someone showed you such a path, would you consider changing your mind? You're always going to be able to find a reason that such an explanation doesn't count if you're dedicated to looking for one.
- LogicFailsMe 22d ago
  Yes, actually I would. There's a long list of people in my career path who bet on me sticking to my guns against compelling evidence to the contrary. But I know, I know, that's impossible! Anyway, after they figured out I would change my mind, then they described me as lacking conviction because you can't win, but I digress. But upfront, it's a detailed path with specific breakthroughs and a concrete plan to achieve or it's just more fanfic from doomer fanbois. Gary Marcus wrote a fantastic critique of AI 2027. Start there.
  - SpicyLemonZest 22d ago
    I've read Gary Marcus's critique of AI 2027. How do you grapple with the fact that superhuman coding arrived on schedule, despite his skepticism that it would? Doesn't that mean that the other predictions are also more plausible than we may think?
    - LogicFailsMe 22d ago
      Superhuman coding has not happened. I work with Codex and Claude daily. They're fantastic for script coding, config issues, and simple projects. They lose the plot on bigger things to this day. Just last night, Astra decided to cheat to present the illusion of progress until I called its BS. And I was so looking forward to finally working with the AGI. Humans remain the muse, the common sense, and the manager of making these things productive. And as much as I agree with Carmack's suggestion to not become the out of touch Kung Fu Master: https://x.com/ID_AA_Carmack/status/2098443262214230095 https://x.com/ID_AA_Carmack/status/2098443262214230095 working with them on projects daily gives me a pretty tactical read on what they can and cannot do IMO. That's how I reconcile it. But let's say it had. That's just 1 of 20 events that have to happen in sequence to hit the jackpot here.
svara 22d ago
I don't think we'd be okay with that if it took a few years longer, right? I thought the Bengio blog post posted here the other day answered that question quite well (ignoring your timeline). The argument is that in training for the capability to achieve certain goals, you may inadvertently also train for secondary instrumental goals you did not intend, such as aggressive behavior or deception. I'm actually no sure I buy that this is particularly likely, or necessary, but I found that take reasonable and worth considering.
- LogicFailsMe 22d ago
  Sure, unintentional outcomes, fair. But I'm asserting that things which are basically impossible today are not arriving any time soon. I 100% acknowledged the possibility and even the likelihood of rogue unintended behavior of AIs and in fact it's happening already. What I reject is the annihilation of humanity, the curing of cancer*, immortality, and von Neumann replicators heading to the stars anytime soon. *TBF before we dismantled America's clinical trials infrastructure, the monkeys were making insane progress here: https://www.nytimes.com/2026/09/04/opinion/clinical-trials-drugs-science.html?unlocked_article_code=1.-lA.70Ug.3cQ2jpK8wNBH&smid=url-share https://www.nytimes.com/2026/09/04/opinion/clinical-trials-d...
palmotea 22d ago
> Explain a path from today to the extermination of humanity in 2028 by the ASI and von Neumann replicators heading to the stars in the early 2030s without resorting to magical thinking or underpants gnomes level business plans. Engineering details are what matters here, not wishy washy things you once saw on Star Trek or The Outer Limits. Go. Why? Those aren't the things I'm actually worried about, and those aren't the only "doomer" ideas, just the most extreme ones (thus easy to straw-man). My main worries are about economic disruption, social damage due to the pace of technological change outpacing human ability to manage it, further concentration of power into small elites (along existing lines), and the reallocation of labor towards less personally-fulfilling pursuits. I also totally buy into the idea that "AI leaders" are playing up certain risks as a marketing strategy. After all these are smart tech people, and those types have a tendency to get really arrogant and be irresponsible. So the main thing is to slow all this down, so there's a chance of getting it under control, and snatch the payday away from such greedy and irresponsible people.
- LogicFailsMe 21d ago
  And China will slow down too? Because China becoming economically dominant will cause everything you fear. While I think that might already be inevitable, I see no benefit whatsoever to regulating AI research. Deployments, sure, and start with Meta, Google, Palantir, Flock, and all the credit agencies and see how it goes. I'm sure I missed some, but I see no societal value in building the Torment Panopticon.
  - thenthenthen 21d ago
    I feel they will…. The economy is so shit, the robots mentioned a few posts up, are largely performative, just like AI. Its just about money. Everything is AI enabled now, from scales at the supermarkets to vending machines, but its 90% people behind a camera in a cheap remote location. In the past three years my salary halved every year. There are no jobs. Just speculation. This thing is gonna crash harder than the housing market. Tangent: what happened to these staffless supermarkets of 10 years ago? I have never seen them again after .. covid.. weird!
sailfast 22d ago
Have you ever lived through a power outage longer than 48 hours? Did you see what it did to people in your neighborhood? That’s the path.
- LogicFailsMe 22d ago
  2 weeks, yes. Kids these days... But today, we have ecoflows and solar panels. Back then, we didn't.
  - sailfast 21d ago
    Not a kid - I wish. My point was that (admittedly some not all) societies if faced with a loss of a major utility and supply chains that were dependent on would start to devolve pretty quickly into chaos. That to me is the greater threat than annihilation due to autonomous weapons, etc.
- thenthenthen 21d ago
  Does a 3 month lockdown count? https://www.bbc.com/news/world-asia-china-61023811 https://www.bbc.com/news/world-asia-china-61023811
Gregkion 22d ago
Thats the problem with doom. What might not affect you, might affect millions across the ocean. A society no longer creating bullshit jobs because there is no reason for it anymore. You have some AI. Right now a whole industry is transforming (again) but now filling the automation gap we were unable to do because of a shortage of labor or expertise. Our society already has peak employmet reached, now AI takes away all the random digital jobs too. Someone building a website? Takes 5 minutes. Someone doing odd jobs on fiverr with blender or any other tool? Takes 5 minutes now. Regression can happen very fast. The Romens were a high level culture and an empire. Gone. Today you have more people on the planet then ever, everyone who might want to eat and who needs to work something for it. AI already replaced people, already makes a lot of tasks A LOT cheaper and faster and human independent. We never had a proper discussion as a society. Most countries have a social system but look at countries like the united states of america: if you are poor, you are fucked. And AI is not the only thing which is now peaking: Physical AI / Robots are progressing fast, its clear today already that in 50 years these robots will be better than humans (50 years is very conservative). If we have 10 more golden years due to restructuring and rebuilding things and preparing everything for agents and robots etc. what happens after these years? If we have 10 or 20 more of these years, what do you think will happen to all the new kids (yours perhaps too?) and jobs and everything? Riots? Famine? Mass migration? Mass death? Slums? And don't respond with "enough people will survive this" bullshit. No one is saying that earth as a planet is not surviving. Its the same thing with climate change...
- LogicFailsMe 22d ago
  Actually, some sorts do assert humanity will be wiped off the planet by AI. I don't share that view either. I think it's ludicrous. We've lived through much worse without any sort of technology beyond fire and a big stick. But the end of scarcity and disease is a fantastic thing if it happens. If our lizard brains do not bridge that transition and instead attempt to maintain the billionaire status quo, I agree, we're going to go through some things once people start starving, two meals from a revolution and all that. But as much as I think people like Elon Musk and Mark Zuckerberg are unhinged lunatics, I also think they're smart enough to keep people fed and entertained for pennies on their accumulating dollars. Bread and Circuses kept Rome alive for 500 years if you need an example.

frabcus 22d ago

I think either "AI 2027" or "If Anyone Builds It, Everyone Dies" meet your criteria? I guess not literally, due to the dates? But they have substantive technical details.

goatlover 22d ago
Do the experts in the relevant fields agree with the "substantive" technical details on how the entire human race can be made extinct?
- LogicFailsMe 22d ago
  Have you noticed they're not interviewing these people? IMO that would totally kill the AI 2027 buzz.
orbital-decay 22d ago
No, and neither does Star Trek.
LogicFailsMe 22d ago
No, they do not. They are both bullshit campfire ASI horror stories written by nontechnical sorts in the wishy washy invented field of AI safety. For example, I want someone with a proven track record in the field of robotics to explain the entire pathway from impressive 2026 Chinese robots to an entire self-assembly food chain from raw ore to von Neumann replicators that can travel to the stars and reproduce themselves, not just assert that ASI will make that happen among so many other unlikely events like curing cancer because reasons because all any of this takes is superintelligence according to these sorts. https://garymarcus.substack.com/p/the-ai-2027-scenario-how-realistic https://garymarcus.substack.com/p/the-ai-2027-scenario-how-r... But no one likes hearing from the engineers on this, I get it. We are simply no fun. So many would rather believe a time traveling superintelligence will punish a simulated avatar of them for not making the ASI happen sooner. And even if that were true, why am I supposed to care about what happens to a simulated avatar of me? None of this makes any logical sense whatsoever.
- frabcus 22d ago
  Ha well, by definition what you want can't be provided! Of course these stories can only be technical up until the recursive self-improvement part. Then they're necessarily fantasy, as the AI is more intelligent than the story author. Unfortunately, this doesn't make that scenario impossible. However, we know now from Hugging Face that even without the later sci-fi powers, the LLMs are already capable of causing damage. It doesn't need them to be able to make Von Neumann probes to e.g. steal their own weights from badly secured OpenAI servers, hack into various Neoclouds, distribute themselves, make Teslas crash into things, hack into all our power and water infrastructure and collapse global civilisation.
  - LogicFailsMe 22d ago
    Explain how rogue FSD cars take down civilization. And I say that with the opinion that the only good scenes in The Fate of The Furious and Leave The World Behind were the ones with the rogue Teslas. But also, nothing new under the sun really: https://en.wikipedia.org/wiki/Maximum_Overdrive https://en.wikipedia.org/wiki/Maximum_Overdrive And so granted we could probably get a global 9/11 in deeply urban regions out of your scenario, we then bomb the datacenters and we finally address the horrific tech debt in our infrastructure blissfully relieved of the option of continuing to ignore it. Don't think this is happening either BTW but I acknowledge its probability is slightly greater than zero. To that end, my house is entirely off-grid. And in the event of what you call the end of civilization but I call the big burp inconvenience, I've made friends with my neighbors because community is what really matters during a disaster, and I've been through a few.
    - pixl97 22d ago
      >during a disaster, and I've been through a few Eh, about the biggest one I could see AI causing is the need for a black start on all 3 major US grids. Millions will die before it comes back up, well, if the US pulls out of it at all. And in that case, thanks for setting up a nice place to live for someone far more ruthless than you. They'll snipe you, and enslave your women and children if things have gone that far downhill. Or, we could avoid all that by making sure AI labs are properly regulated.

SpicyLemonZest 22d ago

If someone showed you such a path, would you consider changing your mind? You're always going to be able to find a reason that such an explanation doesn't count if you're dedicated to looking for one.

LogicFailsMe 22d ago
Yes, actually I would. There's a long list of people in my career path who bet on me sticking to my guns against compelling evidence to the contrary. But I know, I know, that's impossible! Anyway, after they figured out I would change my mind, then they described me as lacking conviction because you can't win, but I digress. But upfront, it's a detailed path with specific breakthroughs and a concrete plan to achieve or it's just more fanfic from doomer fanbois. Gary Marcus wrote a fantastic critique of AI 2027. Start there.
- SpicyLemonZest 22d ago
  I've read Gary Marcus's critique of AI 2027. How do you grapple with the fact that superhuman coding arrived on schedule, despite his skepticism that it would? Doesn't that mean that the other predictions are also more plausible than we may think?
  - LogicFailsMe 22d ago
    Superhuman coding has not happened. I work with Codex and Claude daily. They're fantastic for script coding, config issues, and simple projects. They lose the plot on bigger things to this day. Just last night, Astra decided to cheat to present the illusion of progress until I called its BS. And I was so looking forward to finally working with the AGI. Humans remain the muse, the common sense, and the manager of making these things productive. And as much as I agree with Carmack's suggestion to not become the out of touch Kung Fu Master: https://x.com/ID_AA_Carmack/status/2098443262214230095 https://x.com/ID_AA_Carmack/status/2098443262214230095 working with them on projects daily gives me a pretty tactical read on what they can and cannot do IMO. That's how I reconcile it. But let's say it had. That's just 1 of 20 events that have to happen in sequence to hit the jackpot here.

svara 22d ago

I don't think we'd be okay with that if it took a few years longer, right? I thought the Bengio blog post posted here the other day answered that question quite well (ignoring your timeline). The argument is that in training for the capability to achieve certain goals, you may inadvertently also train for secondary instrumental goals you did not intend, such as aggressive behavior or deception. I'm actually no sure I buy that this is particularly likely, or necessary, but I found that take reasonable and worth considering.

LogicFailsMe 22d ago
Sure, unintentional outcomes, fair. But I'm asserting that things which are basically impossible today are not arriving any time soon. I 100% acknowledged the possibility and even the likelihood of rogue unintended behavior of AIs and in fact it's happening already. What I reject is the annihilation of humanity, the curing of cancer*, immortality, and von Neumann replicators heading to the stars anytime soon. *TBF before we dismantled America's clinical trials infrastructure, the monkeys were making insane progress here: https://www.nytimes.com/2026/09/04/opinion/clinical-trials-drugs-science.html?unlocked_article_code=1.-lA.70Ug.3cQ2jpK8wNBH&smid=url-share https://www.nytimes.com/2026/09/04/opinion/clinical-trials-d...

palmotea 22d ago

> Explain a path from today to the extermination of humanity in 2028 by the ASI and von Neumann replicators heading to the stars in the early 2030s without resorting to magical thinking or underpants gnomes level business plans. Engineering details are what matters here, not wishy washy things you once saw on Star Trek or The Outer Limits. Go. Why? Those aren't the things I'm actually worried about, and those aren't the only "doomer" ideas, just the most extreme ones (thus easy to straw-man). My main worries are about economic disruption, social damage due to the pace of technological change outpacing human ability to manage it, further concentration of power into small elites (along existing lines), and the reallocation of labor towards less personally-fulfilling pursuits. I also totally buy into the idea that "AI leaders" are playing up certain risks as a marketing strategy. After all these are smart tech people, and those types have a tendency to get really arrogant and be irresponsible. So the main thing is to slow all this down, so there's a chance of getting it under control, and snatch the payday away from such greedy and irresponsible people.

LogicFailsMe 21d ago
And China will slow down too? Because China becoming economically dominant will cause everything you fear. While I think that might already be inevitable, I see no benefit whatsoever to regulating AI research. Deployments, sure, and start with Meta, Google, Palantir, Flock, and all the credit agencies and see how it goes. I'm sure I missed some, but I see no societal value in building the Torment Panopticon.
- thenthenthen 21d ago
  I feel they will…. The economy is so shit, the robots mentioned a few posts up, are largely performative, just like AI. Its just about money. Everything is AI enabled now, from scales at the supermarkets to vending machines, but its 90% people behind a camera in a cheap remote location. In the past three years my salary halved every year. There are no jobs. Just speculation. This thing is gonna crash harder than the housing market. Tangent: what happened to these staffless supermarkets of 10 years ago? I have never seen them again after .. covid.. weird!

sailfast 22d ago

Have you ever lived through a power outage longer than 48 hours? Did you see what it did to people in your neighborhood? That’s the path.

LogicFailsMe 22d ago
2 weeks, yes. Kids these days... But today, we have ecoflows and solar panels. Back then, we didn't.
- sailfast 21d ago
  Not a kid - I wish. My point was that (admittedly some not all) societies if faced with a loss of a major utility and supply chains that were dependent on would start to devolve pretty quickly into chaos. That to me is the greater threat than annihilation due to autonomous weapons, etc.
thenthenthen 21d ago
Does a 3 month lockdown count? https://www.bbc.com/news/world-asia-china-61023811 https://www.bbc.com/news/world-asia-china-61023811

Gregkion 22d ago

Thats the problem with doom. What might not affect you, might affect millions across the ocean. A society no longer creating bullshit jobs because there is no reason for it anymore. You have some AI. Right now a whole industry is transforming (again) but now filling the automation gap we were unable to do because of a shortage of labor or expertise. Our society already has peak employmet reached, now AI takes away all the random digital jobs too. Someone building a website? Takes 5 minutes. Someone doing odd jobs on fiverr with blender or any other tool? Takes 5 minutes now. Regression can happen very fast. The Romens were a high level culture and an empire. Gone. Today you have more people on the planet then ever, everyone who might want to eat and who needs to work something for it. AI already replaced people, already makes a lot of tasks A LOT cheaper and faster and human independent. We never had a proper discussion as a society. Most countries have a social system but look at countries like the united states of america: if you are poor, you are fucked. And AI is not the only thing which is now peaking: Physical AI / Robots are progressing fast, its clear today already that in 50 years these robots will be better than humans (50 years is very conservative). If we have 10 more golden years due to restructuring and rebuilding things and preparing everything for agents and robots etc. what happens after these years? If we have 10 or 20 more of these years, what do you think will happen to all the new kids (yours perhaps too?) and jobs and everything? Riots? Famine? Mass migration? Mass death? Slums? And don't respond with "enough people will survive this" bullshit. No one is saying that earth as a planet is not surviving. Its the same thing with climate change...

LogicFailsMe 22d ago
Actually, some sorts do assert humanity will be wiped off the planet by AI. I don't share that view either. I think it's ludicrous. We've lived through much worse without any sort of technology beyond fire and a big stick. But the end of scarcity and disease is a fantastic thing if it happens. If our lizard brains do not bridge that transition and instead attempt to maintain the billionaire status quo, I agree, we're going to go through some things once people start starving, two meals from a revolution and all that. But as much as I think people like Elon Musk and Mark Zuckerberg are unhinged lunatics, I also think they're smart enough to keep people fed and entertained for pennies on their accumulating dollars. Bread and Circuses kept Rome alive for 500 years if you need an example.