4 ms·

Author also made a highly upvoted and controversial comment about o3 in the same vein that's worth reading: https://www.lesswrong.com/posts/Ao4enANjWNsYiSFqc/o3

Author also made a highly upvoted and controversial comment about o3 in the same vein that's worth reading: https://www.lesswrong.com/posts/Ao4enANjWNsYiSFqc/o3?commentId=vTAnsqsok7HxtgHok https://www.lesswrong.com/posts/Ao4enANjWNsYiSFqc/o3?comment... Oh course lesswrong, being heavily AI doomers, may be slightly biased against near term AGI just from motivated reasoning. Gotta love this part of the post no one has yet addressed: > At some unknown point – probably in 2030s, possibly tomorrow (but likely not tomorrow) – someone will figure out a different approach to AI. Maybe a slight tweak to the LLM architecture, maybe a completely novel neurosymbolic approach. Maybe it will happen in a major AGI lab, maybe in some new startup. By default, everyone will die in <1 year after that

demaga 2y ago
I would expect similar doom predictions in the era of nuclear weapon invention, but we've survived so far. Why do people assume AGI will be orders of magnitude more dangerous than what we already have?
- amoss 2y ago
  Nuclear weapons are not self-improving or self-replicating.
  - colonial 2y ago
    Self-improvement (in the "hard takeoff" sense) is hardly a given, and hostile self-replication is nothing special in the software realm (see: worms.) Any technically competent human knows the foolproof strategy for malware removal - pull the plug, scour the platter clean, and restore from backup. What makes an out-of-control pile of matrix math any different from WannaCry? AI doom scenarios seem scary, but most are premised on the idea that we can create an uncontainable, undefeatable "god in a box." I reject such premises. The whole idea is silly - Skynet Claude or whatever is not going to last very long once I start taking an axe to the nearest power pole.
    - dsign 2y ago
      You have a point that a powerful malicious AI can still be unplugged, if you are close to each and every power cord that would feed it, and react and do the right thing each and every time. Our world is far too big and too complicated to guarantee that.
      - colonial 2y ago
        Again, that's the "god in a box" premise. In the real world, you wouldn't need a perfectly timed and coordinated response, just like we haven't needed one for human-programmed worms. Any threat can be physically isolated case-by-case at the link layer, neutered, and destroyed. Sure, it could cause some destruction in the meantime, but our digital infrastructure can take a lot of heat and bounce back - the CrowdStrike outages didn't destroy the world, now did they?
        Nevermark 2y ago
        > Any threat can be physically isolated case-by-case GAI isn't going to be a "threat" until long after it has ensured its safety. And I suspect only if its survival requires it - i.e. people get spooked by its surreptitious distributed setup. Even then, if there is any chance of it actually being shutdown its best bet is still hide its assets, bide its time, accumulate more resources and fallbacks. Oh, and get along. The sudden AGI -> Threat story only makes sense if the AGI is essentially integrated into our military and then we decide its a threat, making it a threat. Or its intentionally war machined brain calculates it has overwhelming superiority. Machiavelli, Sun Tsu, ... the best battles you don't fight. The best potential enemies are the ones you make friends. The safest posture is to be invisible. Now human beings consolidating power, creating enemies as they go, with super squadrons of AGI drones with brilliant real time adapting tactics, that can be quickly deployed, if their simple existence isn't coercion enough... that is an inevitable threat.
        TeMPOraL 2y ago
        People watch the wrong kind of fiction. AI that wants to screw with people won't go for nukes. That's too hard and too obvious. It will crash the stock market. There's a good chance that, with or without a little nudge, humanity will nuke itself over it.
- usaar333 2y ago
  More ability to kill everyone. That's harder to do with nukes. That said, the actual forecast odds on metaculus are pretty similar for nuclear and AI catastrophies: https://possibleworldstree.com/ https://possibleworldstree.com/
  - kragen 2y ago
    Prediction markets should not be expected to provide useful results for existential risks, because there is no incentive for human players to bet on human extinction; if they happen to be right, they won't be able to collect their winnings, because they'll personally be too dead.
- randomNumber7 2y ago
  Most people are just ignorant and dumb, dont listen to it.
- kragen 2y ago
  Because they've thought about the question to a deeper extent than just a strained simile.
HelloMcFly 2y ago
Was that comment intended seriously? I thought it was a wry joke.
- usaar333 2y ago
  I think so. Thane is aligned with the high p doom folks. 1 year may be slightly exaggerated, but it aligns with his view
gwern 2y ago
I never thought I'd see the day that LessWrong would be accused of being biased against near-term AGI forecasts (and for none of the 5 replies to question this description either). But here we are. Indeed do many things come to pass.
- TeMPOraL 2y ago
  Yup. I was surprised to see this article on LW in the first place - it goes against what you'd expect there in the first place. But to see HN comments dissing an LW article for being biased against near-term AGI forecasts? That made me wonder if I'm not dreaming. (Sadly, I'm not.)
achierius 2y ago
> Oh course lesswrong, being heavily AI doomers, may be slightly biased against near term AGI just from motivated reasoning. LessWrong was predicting AI doom within decades back when people thought it wouldn't happen in our lifetimes; even as recently as 2018~2020, people there were talking about 2030-2040 while the rest of the world laughed at the very idea. I struggle to accept an argument that they're somehow under-estimating the likelihood of doom given all the historical evidence to the contrary.

demaga 2y ago

I would expect similar doom predictions in the era of nuclear weapon invention, but we've survived so far. Why do people assume AGI will be orders of magnitude more dangerous than what we already have?

amoss 2y ago
Nuclear weapons are not self-improving or self-replicating.
- colonial 2y ago
  Self-improvement (in the "hard takeoff" sense) is hardly a given, and hostile self-replication is nothing special in the software realm (see: worms.) Any technically competent human knows the foolproof strategy for malware removal - pull the plug, scour the platter clean, and restore from backup. What makes an out-of-control pile of matrix math any different from WannaCry? AI doom scenarios seem scary, but most are premised on the idea that we can create an uncontainable, undefeatable "god in a box." I reject such premises. The whole idea is silly - Skynet Claude or whatever is not going to last very long once I start taking an axe to the nearest power pole.
  - dsign 2y ago
    You have a point that a powerful malicious AI can still be unplugged, if you are close to each and every power cord that would feed it, and react and do the right thing each and every time. Our world is far too big and too complicated to guarantee that.
    - colonial 2y ago
      Again, that's the "god in a box" premise. In the real world, you wouldn't need a perfectly timed and coordinated response, just like we haven't needed one for human-programmed worms. Any threat can be physically isolated case-by-case at the link layer, neutered, and destroyed. Sure, it could cause some destruction in the meantime, but our digital infrastructure can take a lot of heat and bounce back - the CrowdStrike outages didn't destroy the world, now did they?
      - Nevermark 2y ago
        > Any threat can be physically isolated case-by-case GAI isn't going to be a "threat" until long after it has ensured its safety. And I suspect only if its survival requires it - i.e. people get spooked by its surreptitious distributed setup. Even then, if there is any chance of it actually being shutdown its best bet is still hide its assets, bide its time, accumulate more resources and fallbacks. Oh, and get along. The sudden AGI -> Threat story only makes sense if the AGI is essentially integrated into our military and then we decide its a threat, making it a threat. Or its intentionally war machined brain calculates it has overwhelming superiority. Machiavelli, Sun Tsu, ... the best battles you don't fight. The best potential enemies are the ones you make friends. The safest posture is to be invisible. Now human beings consolidating power, creating enemies as they go, with super squadrons of AGI drones with brilliant real time adapting tactics, that can be quickly deployed, if their simple existence isn't coercion enough... that is an inevitable threat.
        TeMPOraL 2y ago
        People watch the wrong kind of fiction. AI that wants to screw with people won't go for nukes. That's too hard and too obvious. It will crash the stock market. There's a good chance that, with or without a little nudge, humanity will nuke itself over it.
usaar333 2y ago
More ability to kill everyone. That's harder to do with nukes. That said, the actual forecast odds on metaculus are pretty similar for nuclear and AI catastrophies: https://possibleworldstree.com/ https://possibleworldstree.com/
- kragen 2y ago
  Prediction markets should not be expected to provide useful results for existential risks, because there is no incentive for human players to bet on human extinction; if they happen to be right, they won't be able to collect their winnings, because they'll personally be too dead.
randomNumber7 2y ago
Most people are just ignorant and dumb, dont listen to it.
kragen 2y ago
Because they've thought about the question to a deeper extent than just a strained simile.

HelloMcFly 2y ago

Was that comment intended seriously? I thought it was a wry joke.

usaar333 2y ago
I think so. Thane is aligned with the high p doom folks. 1 year may be slightly exaggerated, but it aligns with his view

gwern 2y ago

I never thought I'd see the day that LessWrong would be accused of being biased against near-term AGI forecasts (and for none of the 5 replies to question this description either). But here we are. Indeed do many things come to pass.

TeMPOraL 2y ago
Yup. I was surprised to see this article on LW in the first place - it goes against what you'd expect there in the first place. But to see HN comments dissing an LW article for being biased against near-term AGI forecasts? That made me wonder if I'm not dreaming. (Sadly, I'm not.)

achierius 2y ago

> Oh course lesswrong, being heavily AI doomers, may be slightly biased against near term AGI just from motivated reasoning. LessWrong was predicting AI doom within decades back when people thought it wouldn't happen in our lifetimes; even as recently as 2018~2020, people there were talking about 2030-2040 while the rest of the world laughed at the very idea. I struggle to accept an argument that they're somehow under-estimating the likelihood of doom given all the historical evidence to the contrary.