3 ms·

> Many of them also expect that, without heroic effort, AGI development will lead to human extinction. Odd to me. Not biological warfare? Global warming? All-o

> Many of them also expect that, without heroic effort, AGI development will lead to human extinction. Odd to me. Not biological warfare? Global warming? All-out nuclear war? I guess The Terminator was a formative experience for them. (For me perhaps it was The Andromeda Strain.)

mitthrowaway2 1y ago
These aren't mutually exclusive. Even in The Terminator, Skynet's method of choice is nuclear war. Yudkowsky frequency expressses concern that a malevolent AI might synthesize a bioweapon. I personally worry that destroying the ozone layer might be an easy opening volley. Either way, I don't want a really smart computer spending its time figuring out plans to end the human species, because I think there are too many ways to be successful.
- staticman2 1y ago
  Terminator descends from a tradition of science fiction cold war parables. Even in Terminator 2 there's a line suggesting the movie isn't really about robots: John:We're not gonna make it, are we? People, I mean. Terminator: It's in your nature to destroy yourselves. Seems odd to worry about computers shooting the ozone when there's plenty of real existential threats loaded in missles aimed at you right now.
  - mitthrowaway2 1y ago
    I'm not in any way discounting the danger represented by those missiles. In fact I think AI only makes it more likely that they might someday be launched. But I will say that in my experience the error-condition that causes a system to fail is usually the one that didn't seem likely to happen, because the more obvious failure modes were taken seriously from the beginning. Is it so unusual to be able to consider more than one risk at a time?
myaccountonhn 1y ago
That's what was so strange with EA and rationalist movements. A highly theoretical model that AGI could wipe us all out vs the very real issue of global warming and pretty much all emphasis was on AGI.
- wredcoll 1y ago
  Agi is a lot more fun to worry about and asks a lot less of you. Sort of like advocating for the "unborn" vs veterans/homeless/addicts.
eschaton 1y ago
It makes a lot of sense when you realize that for many of the “leaders” in this community like Yudkowsky, their understanding of science (what it is, how it works, and its potential) comes entirely from reading science fiction and playing video games. Sad because Eli’s dad was actually a real and well-credentialed researcher at Bell Labs. Too bad he let his son quit school at an early age to be an autodidact.
- elliotto 1y ago
  I'm not at all a rationalist or a defender, but big yud has an epistemology that takes the form of the rationalist sacred text mentioned in the article (the sequences). A lot of it is well thought out, and probably can't be discarded as just coming from science fiction and video games. Yud has a great 4 hour talk with Stephen Wolfram where he holds his own.
  - eschaton 1y ago
    Holding one’s own against Stephen Wolfram isn’t exactly the endorsement it might seem.
    - elliotto 1y ago
      I'm interested in this perspective, I haven't come across much criticism of Wolfram but I haven't really looked for it much either. Is it because of his theory of everything ruliad stuff? I really enjoy his blog posts and his work on automata seems to be well respected. I've felt he presents a solid epistemology.
      - eschaton 1y ago
        He’s pretty widely regarded as a kook by real scientists. He did some good math and physics back in the day and then some good work on cellular automata, and assumed that meant he was smarter than everybody else and that all of science was in fact a subset of his special focus, which it’s pretty trivially not. Furthermore he has a very long history of grossly misrepresenting his accomplishments, whether by claiming authorship of his employees’ accomplishments or denying he was ever exposed to work that he built upon. For example, I believe it’s been proven now that he had substantial exposure to MACSYMA before he started to lead his employees in the development of Mathematica—but that didn’t stop him from claiming it was entirely original and even threatening others with patent and other IP infringement claims for trying to do things that were similar. (Which is the other problem with him: He doesn’t just take credit for the work he pays others to do, he’s an “IP protection” maximalist.)
bell-cot 1y ago
My interpretation: When they say "will lead to human extinction", they are trying to vocalize their existential terror that an AGI would render them and their fellow rationalist cultists permanently irrelevant - by being obviously superior to them, by the only metric that really matters to them.
- lostmsu 1y ago
  You sound like you wouldn't feel existential terror if after typing "My interpretation: " into the text field you'd see the rest of your message suggested by Copilot exactly how you wrote it letter by letter. And the same in every other conversation. How about people interrupting you in "real" life interaction after an AI predicted your whole tirade for them and they read it faster than you said it, and also read an analysis of it? Dystopian sci-fi for sure, but many people dismissing LLMs as not AGI do so because LLMs are just "token predictors".
  - bell-cot 1y ago
    Scroll up and read the comment by JKCalhoun, for the context of my prior comment. Or: I'm decades too old to have grown up in the modern "failure to get attention online = death" social media dystopia. A dozen lines of shell script could pretty well predict the meals I eat in a day. Or how many times I get up to pee in the night. Neither of those facts bother me. And if I want some "the world is better for my being here" feedback - there are a dozen or more local non-profits, churches, and frail old friends/neighbors who would would happily welcome my assistance or visit on any given day.
DonsDiscountGas 1y ago
Check out "the precipice" by Tony Ord. Biological warfare and global warming are unlikely to lead to total human extinction (though both present large risks of massive harm). Part of the argument is that we've had nuclear weapons for a long time but no apocalypse so the annual risk can't be larger than 1%, whereas we've never created AI so it might be substantially larger. Not a rock solid argument obviously, but we're dealing with a lot of unknowns. A better argument is that most of those other risks are not neglected, plenty of smart people working against nuclear war. Whereas (up until a few years ago) very few people considered AI a real threat, so the marginal benefit of a new person working on it should be bigger.
meowface 1y ago
Most in the community consider nuclear and biological threats to be dire. Many just consider existential threats from AI to be even more probable and damaging.
skybrian 1y ago
Yes, sufficiently high intelligence is sometimes assumed to allow for rapid advances in many scientific areas. So, it could be biological warfare because AGI. Or nanotech, drone warfare, or something stranger. I'm a little skeptical (there may be bottlenecks that can't be solved by thinking harder), but I don't see how it can be ruled out.
rsynnott 1y ago
I mean, this is the religion/philosophy which produced Roko's Basilisk (and not one of their weird offshoot murder-cults, either, it showed up on LessWrong, and was taken at least somewhat seriously by people there, to the point that Yudkowsky censored it. Their beliefs about AI are... out there.
- ben_w 1y ago
  > and was taken at least somewhat seriously by people there, to the point that Yudkowsky censored it. Roko isn't taken seriously. What was taken seriously is ~ "if you've had an idea that you yourself think will harm people to even know about it, don't share it".
  - rsynnott 1y ago
    But if you think that knowing about the basilisk is harmful to people who don't know about it, surely you are taking it at least somewhat seriously?
    - ben_w 1y ago
      Not necessarily, it also comes from taking your users seriously before you even know what they were saying. Consider: If you trust the people on your forum to be sensible, and one day one of your users writes: "I have realised something which is dangerous to know. Here it is: …" then the sensible choice is to (1) stop reading immediately, don't even find out what's after the ellipsis, (2) delete/hide/otherwise remove their message, and (3) put in a general statement of "don't write things that you think are dangerous for people to know".

mitthrowaway2 1y ago

These aren't mutually exclusive. Even in The Terminator, Skynet's method of choice is nuclear war. Yudkowsky frequency expressses concern that a malevolent AI might synthesize a bioweapon. I personally worry that destroying the ozone layer might be an easy opening volley. Either way, I don't want a really smart computer spending its time figuring out plans to end the human species, because I think there are too many ways to be successful.

staticman2 1y ago
Terminator descends from a tradition of science fiction cold war parables. Even in Terminator 2 there's a line suggesting the movie isn't really about robots: John:We're not gonna make it, are we? People, I mean. Terminator: It's in your nature to destroy yourselves. Seems odd to worry about computers shooting the ozone when there's plenty of real existential threats loaded in missles aimed at you right now.
- mitthrowaway2 1y ago
  I'm not in any way discounting the danger represented by those missiles. In fact I think AI only makes it more likely that they might someday be launched. But I will say that in my experience the error-condition that causes a system to fail is usually the one that didn't seem likely to happen, because the more obvious failure modes were taken seriously from the beginning. Is it so unusual to be able to consider more than one risk at a time?

myaccountonhn 1y ago

That's what was so strange with EA and rationalist movements. A highly theoretical model that AGI could wipe us all out vs the very real issue of global warming and pretty much all emphasis was on AGI.

wredcoll 1y ago
Agi is a lot more fun to worry about and asks a lot less of you. Sort of like advocating for the "unborn" vs veterans/homeless/addicts.

eschaton 1y ago

It makes a lot of sense when you realize that for many of the “leaders” in this community like Yudkowsky, their understanding of science (what it is, how it works, and its potential) comes entirely from reading science fiction and playing video games. Sad because Eli’s dad was actually a real and well-credentialed researcher at Bell Labs. Too bad he let his son quit school at an early age to be an autodidact.

elliotto 1y ago
I'm not at all a rationalist or a defender, but big yud has an epistemology that takes the form of the rationalist sacred text mentioned in the article (the sequences). A lot of it is well thought out, and probably can't be discarded as just coming from science fiction and video games. Yud has a great 4 hour talk with Stephen Wolfram where he holds his own.
- eschaton 1y ago
  Holding one’s own against Stephen Wolfram isn’t exactly the endorsement it might seem.
  - elliotto 1y ago
    I'm interested in this perspective, I haven't come across much criticism of Wolfram but I haven't really looked for it much either. Is it because of his theory of everything ruliad stuff? I really enjoy his blog posts and his work on automata seems to be well respected. I've felt he presents a solid epistemology.
    - eschaton 1y ago
      He’s pretty widely regarded as a kook by real scientists. He did some good math and physics back in the day and then some good work on cellular automata, and assumed that meant he was smarter than everybody else and that all of science was in fact a subset of his special focus, which it’s pretty trivially not. Furthermore he has a very long history of grossly misrepresenting his accomplishments, whether by claiming authorship of his employees’ accomplishments or denying he was ever exposed to work that he built upon. For example, I believe it’s been proven now that he had substantial exposure to MACSYMA before he started to lead his employees in the development of Mathematica—but that didn’t stop him from claiming it was entirely original and even threatening others with patent and other IP infringement claims for trying to do things that were similar. (Which is the other problem with him: He doesn’t just take credit for the work he pays others to do, he’s an “IP protection” maximalist.)

bell-cot 1y ago

My interpretation: When they say "will lead to human extinction", they are trying to vocalize their existential terror that an AGI would render them and their fellow rationalist cultists permanently irrelevant - by being obviously superior to them, by the only metric that really matters to them.

lostmsu 1y ago
You sound like you wouldn't feel existential terror if after typing "My interpretation: " into the text field you'd see the rest of your message suggested by Copilot exactly how you wrote it letter by letter. And the same in every other conversation. How about people interrupting you in "real" life interaction after an AI predicted your whole tirade for them and they read it faster than you said it, and also read an analysis of it? Dystopian sci-fi for sure, but many people dismissing LLMs as not AGI do so because LLMs are just "token predictors".
- bell-cot 1y ago
  Scroll up and read the comment by JKCalhoun, for the context of my prior comment. Or: I'm decades too old to have grown up in the modern "failure to get attention online = death" social media dystopia. A dozen lines of shell script could pretty well predict the meals I eat in a day. Or how many times I get up to pee in the night. Neither of those facts bother me. And if I want some "the world is better for my being here" feedback - there are a dozen or more local non-profits, churches, and frail old friends/neighbors who would would happily welcome my assistance or visit on any given day.

DonsDiscountGas 1y ago

Check out "the precipice" by Tony Ord. Biological warfare and global warming are unlikely to lead to total human extinction (though both present large risks of massive harm). Part of the argument is that we've had nuclear weapons for a long time but no apocalypse so the annual risk can't be larger than 1%, whereas we've never created AI so it might be substantially larger. Not a rock solid argument obviously, but we're dealing with a lot of unknowns. A better argument is that most of those other risks are not neglected, plenty of smart people working against nuclear war. Whereas (up until a few years ago) very few people considered AI a real threat, so the marginal benefit of a new person working on it should be bigger.

meowface 1y ago

Most in the community consider nuclear and biological threats to be dire. Many just consider existential threats from AI to be even more probable and damaging.

skybrian 1y ago

Yes, sufficiently high intelligence is sometimes assumed to allow for rapid advances in many scientific areas. So, it could be biological warfare because AGI. Or nanotech, drone warfare, or something stranger. I'm a little skeptical (there may be bottlenecks that can't be solved by thinking harder), but I don't see how it can be ruled out.

rsynnott 1y ago

I mean, this is the religion/philosophy which produced Roko's Basilisk (and not one of their weird offshoot murder-cults, either, it showed up on LessWrong, and was taken at least somewhat seriously by people there, to the point that Yudkowsky censored it. Their beliefs about AI are... out there.

ben_w 1y ago
> and was taken at least somewhat seriously by people there, to the point that Yudkowsky censored it. Roko isn't taken seriously. What was taken seriously is ~ "if you've had an idea that you yourself think will harm people to even know about it, don't share it".
- rsynnott 1y ago
  But if you think that knowing about the basilisk is harmful to people who don't know about it, surely you are taking it at least somewhat seriously?
  - ben_w 1y ago
    Not necessarily, it also comes from taking your users seriously before you even know what they were saying. Consider: If you trust the people on your forum to be sensible, and one day one of your users writes: "I have realised something which is dangerous to know. Here it is: …" then the sensible choice is to (1) stop reading immediately, don't even find out what's after the ellipsis, (2) delete/hide/otherwise remove their message, and (3) put in a general statement of "don't write things that you think are dangerous for people to know".