5 ms·

Regardless of how realistic you think AGI risk is, it’s not marketing hype- it has been a human fear and central in scifi for a long time. Movies like 2001, Ter

by _yb2s 2y ago

Regardless of how realistic you think AGI risk is, it’s not marketing hype- it has been a human fear and central in scifi for a long time. Movies like 2001, Terminator, and the Matrix were hugely popular long before AI was profitable. Arguably ancient myths about golems and genies are essentially the same fear and concept. AI companies are afraid of public fear over AI as it might lead to regulating them out of existence…. They are actively creating marketing hype in the opposite direction, to convince people that AI is safe and useful.

daveguy 2y ago
Fear of AGI is very real. But our proximity to AGI is marketing hype. In terms of existential crisis #2 and #3 are serious and already worth investing in mitigation. But #1 (AGI) is science fiction.
- _yb2s 2y ago
  Fiction is much more than just entertainment... science fiction has been pivotal in both shaping and preparing our society for changes caused by technology. Much of the best science fiction has been so prescient that it's hard for modern audiences to even understand that these things were written before the world was like this. When I share a lot of old sci-fi with my young son, he finds it unremarkable, as they seem to simply be mundane stories about our present reality. Authors like Vernor Vinge wrote about AGI risk in stories because they were personally worried about it, and trying to share something they felt was important with others. You can easily disagree with the warnings and fears shared in particular sci-fi, but to dismiss fiction as categorically irrelevant to our reality is just ignorant. There is some important history here your comments suggest you are unaware of. One reason AI companies talk so much about AI risk despite it being bad for their bottom line isn't marketing hype, it's because many people in those companies are genuinely afraid. One could argue that this is because many of them have been exposed to the rationalist community- which could be (uncharitably?) seen as a doomsday cult obsessed with AGI risk. The founders of many AI startups including OpenAI were heavily influenced and active in this community.
  - nullstyle 2y ago
    Out of curiosity, could you recommend any Science Fiction written before the nuclear age that talks about nuclear proliferation? Or to take something from our present world, what Science Fiction stories would you recommend to talk about the effect of drone tech on trade and shipping?
    - _yb2s 2y ago
      The unintended consequences of new powerful weapons falling into the wrong hands and new trade technology are very old themes in fiction even if the specific technical details are not.
      - nullstyle 2y ago
        I noticed you edited your initial response for tone before I could respond. Regardless, I'll respond to your initial statement to say that no, I'm not trying to make a snarky rebuttal. I'm trying to understand your statements better because my own personal experience and opinions disagree with what you claim and I'd like to integrate your perspective. I don't give a shit about internet points. You specifically mention Vernor Vinge, whom I haven't yet read. My understanding is his work is dependent upon their being an intelligence explosion with the development of AGI, which seems incredibly unlikely given how AI technology has developed thus far. I think we've seen amazing growth in capability but nothing close to an intelligence explosion as I've seen portrayed in SF. As someone working in this space, what lessons would I learn by reading Vinge's work as opposed to spending that time reading more about the history and happenings in the real world of computer security? Now, to respond to your post-edit: I feel like my disconnect with your positions in this thread are around the importance of fiction. I think fiction is a minor muscle in the body that drives technology, if you'll allow the metaphor, whereas you claim "science fiction has been pivotal in both shaping and preparing our society for changes caused by technology." Perhaps you could explain what you mean more concretely? Another confusion I have: The claim of prescience and that the themes are very old seem to be at odds. If I develop an understanding of the dangers of runaway technology through a story about Genies, why do I need more modern work? What if I instead gain these understanding through actual history? The real world has so many more wrinkles than a story and so many more lessons to give about technological development than a story that I balk at words like pivotal. There is not pivot point, only hundreds and hundreds of articulations. Anyways, I've written enough for now, and I don't expect you to follow up.
- mhardcastle 2y ago
  In 2022, the median ML researcher surveyed thought that there is a 5% or 10% chance of AI leading to "human extinction or similarly permanent and severe disempowerment of the human species," depending on how the question was asked. https://aiimpacts.org/2022-expert-survey-on-progress-in-ai/#Existential_risk https://aiimpacts.org/2022-expert-survey-on-progress-in-ai/#...
  - Der_Einzige 2y ago
    I claim in 2024 this has doubled or tripled.
krainboltgreene 2y ago
None of the movies and stories you mentioned were about the "rise of AGI". The closest you can get might be The Matrix via Animatrix? You can go look up what the authors meant by their pieces, there's plenty of analysis of each of those stories. In addition I would love a citation on "the rise of AGI" being a fear in sci-fi for a long time, I've enjoyed quite a bit of science fiction literature and I can only think of a handful that even touched on the idea.
- flir 2y ago
  Neuromancer. I Have No Mouth and I Must Scream. Erewhon.
  - krainboltgreene 2y ago
    You have massively confused "AGI as a plot point" with "is what the literary work is about". For example Neuromancer is about trauma and identity. If I'm being charitable I could give you Erewhon, but that's only because a major theme is anthropocentrism. A vast majority of the book is a criticism of victorian society. Ellison's analysis of his own work is pretty well known and while not specifically about AGI is definitely about technology misuse, so you win there.
    - _yb2s 2y ago
      This is an absurd level of nitpicking... the phrase "is about" means both of those things in common speech, and everyone can tell which you mean depending on context. Language is defined by regular people through its use and context, scholars only describe it after the fact- to their extreme dismay. You might be technically correct within the jargon of, e.g. an academic literature course, but you aren't correct to insist the words mean that unless you are in that context, with that audience, where it has that meaning. Moreover, I don't think you are correct, even in the literary sense, when saying, for example, what Neuromancer is about. I don't really care what an author says it's about, most artists are channeling something unconscious and often have very little conscious understanding of what their work is actually about. I've read plenty of Gibson books. The guy is an absolute luddite, and writes on a typewriter. He is terrified of how dangerous and dehumanizing technology is, and all of his books are "about" an all-consuming fear of technology and the changes it causes.
feoren 2y ago
Spiders have been a human fear for two million years, but I'm not too worried about them causing our downfall as a species. Just because we're afraid of AI doesn't mean it's actually worth fearing.
- _yb2s 2y ago
  Yes, even if something is worth fearing, it's not simply because people are afraid of it already. Did something I write make you think I was arguing otherwise? As an aside, I would argue that this deep rooted human fear of AI is related to the general ancient fear of humans being wiped out by other bands of more technologically advanced humans. Which for most of human history has been the single biggest danger by far. People also have deep instinctual fears of things like spiders and snakes for the same reason- they were serious hazards in the environment where humans evolved. One could say we have evolved these instincts to fear them, because they are worth fearing... not the other way around.
  - feoren 2y ago
    > Did something I write make you think I was arguing otherwise? At the risk of having boring meta-commentary about comments, I feel like I see people often forgetting that their comments are in the context of a larger conversation and are going to be read as responses to the parent comment that they are, well, responding to. And then they get confused when their comments are interpreted in that context, rather than as completely standalone statements. So let's backtrack: (A) AGI is a threat to human civilization. (B) Actually, its risk is marketing hype. (You) It's not marketing hype. Our fear is deep-seated. It's not that weird to interpret the 3rd comment as disagreeing that the fear is just hype. When in fact, you were disagreeing only with the word "marketing". It's "innate" hype, not "marketing" hype. That's a much -- for lack of a better word -- weaker comment, because it is nitpicking about a minor word choice ("it's marketing hype") rather than the sentiment of the comment ("it's just hype"). So now you're asking "why did you assume my comment was interesting, and not just a little nitpick!?" > One could say we have evolved these instincts to fear them, because they are worth fearing But now you're back to saying AI is worth fearing, right? Isn't that what you're telling me I was wrong to assume you're saying? This feels like a needlessly difficult conversation. All that aside, your additions to the conversation are interesting. I actually really like your take on it being a fear of technologically superior tribes. It just doesn't feel like it needs to be quite so adversarial. You established an adversarial attitude with "it's not marketing hype".

by _yb2s 2y ago

daveguy 2y ago

Fear of AGI is very real. But our proximity to AGI is marketing hype. In terms of existential crisis #2 and #3 are serious and already worth investing in mitigation. But #1 (AGI) is science fiction.

_yb2s 2y ago
Fiction is much more than just entertainment... science fiction has been pivotal in both shaping and preparing our society for changes caused by technology. Much of the best science fiction has been so prescient that it's hard for modern audiences to even understand that these things were written before the world was like this. When I share a lot of old sci-fi with my young son, he finds it unremarkable, as they seem to simply be mundane stories about our present reality. Authors like Vernor Vinge wrote about AGI risk in stories because they were personally worried about it, and trying to share something they felt was important with others. You can easily disagree with the warnings and fears shared in particular sci-fi, but to dismiss fiction as categorically irrelevant to our reality is just ignorant. There is some important history here your comments suggest you are unaware of. One reason AI companies talk so much about AI risk despite it being bad for their bottom line isn't marketing hype, it's because many people in those companies are genuinely afraid. One could argue that this is because many of them have been exposed to the rationalist community- which could be (uncharitably?) seen as a doomsday cult obsessed with AGI risk. The founders of many AI startups including OpenAI were heavily influenced and active in this community.
- nullstyle 2y ago
  Out of curiosity, could you recommend any Science Fiction written before the nuclear age that talks about nuclear proliferation? Or to take something from our present world, what Science Fiction stories would you recommend to talk about the effect of drone tech on trade and shipping?
  - _yb2s 2y ago
    The unintended consequences of new powerful weapons falling into the wrong hands and new trade technology are very old themes in fiction even if the specific technical details are not.
    - nullstyle 2y ago
      I noticed you edited your initial response for tone before I could respond. Regardless, I'll respond to your initial statement to say that no, I'm not trying to make a snarky rebuttal. I'm trying to understand your statements better because my own personal experience and opinions disagree with what you claim and I'd like to integrate your perspective. I don't give a shit about internet points. You specifically mention Vernor Vinge, whom I haven't yet read. My understanding is his work is dependent upon their being an intelligence explosion with the development of AGI, which seems incredibly unlikely given how AI technology has developed thus far. I think we've seen amazing growth in capability but nothing close to an intelligence explosion as I've seen portrayed in SF. As someone working in this space, what lessons would I learn by reading Vinge's work as opposed to spending that time reading more about the history and happenings in the real world of computer security? Now, to respond to your post-edit: I feel like my disconnect with your positions in this thread are around the importance of fiction. I think fiction is a minor muscle in the body that drives technology, if you'll allow the metaphor, whereas you claim "science fiction has been pivotal in both shaping and preparing our society for changes caused by technology." Perhaps you could explain what you mean more concretely? Another confusion I have: The claim of prescience and that the themes are very old seem to be at odds. If I develop an understanding of the dangers of runaway technology through a story about Genies, why do I need more modern work? What if I instead gain these understanding through actual history? The real world has so many more wrinkles than a story and so many more lessons to give about technological development than a story that I balk at words like pivotal. There is not pivot point, only hundreds and hundreds of articulations. Anyways, I've written enough for now, and I don't expect you to follow up.
mhardcastle 2y ago
In 2022, the median ML researcher surveyed thought that there is a 5% or 10% chance of AI leading to "human extinction or similarly permanent and severe disempowerment of the human species," depending on how the question was asked. https://aiimpacts.org/2022-expert-survey-on-progress-in-ai/#Existential_risk https://aiimpacts.org/2022-expert-survey-on-progress-in-ai/#...
- Der_Einzige 2y ago
  I claim in 2024 this has doubled or tripled.

krainboltgreene 2y ago

None of the movies and stories you mentioned were about the "rise of AGI". The closest you can get might be The Matrix via Animatrix? You can go look up what the authors meant by their pieces, there's plenty of analysis of each of those stories. In addition I would love a citation on "the rise of AGI" being a fear in sci-fi for a long time, I've enjoyed quite a bit of science fiction literature and I can only think of a handful that even touched on the idea.

flir 2y ago
Neuromancer. I Have No Mouth and I Must Scream. Erewhon.
- krainboltgreene 2y ago
  You have massively confused "AGI as a plot point" with "is what the literary work is about". For example Neuromancer is about trauma and identity. If I'm being charitable I could give you Erewhon, but that's only because a major theme is anthropocentrism. A vast majority of the book is a criticism of victorian society. Ellison's analysis of his own work is pretty well known and while not specifically about AGI is definitely about technology misuse, so you win there.
  - _yb2s 2y ago
    This is an absurd level of nitpicking... the phrase "is about" means both of those things in common speech, and everyone can tell which you mean depending on context. Language is defined by regular people through its use and context, scholars only describe it after the fact- to their extreme dismay. You might be technically correct within the jargon of, e.g. an academic literature course, but you aren't correct to insist the words mean that unless you are in that context, with that audience, where it has that meaning. Moreover, I don't think you are correct, even in the literary sense, when saying, for example, what Neuromancer is about. I don't really care what an author says it's about, most artists are channeling something unconscious and often have very little conscious understanding of what their work is actually about. I've read plenty of Gibson books. The guy is an absolute luddite, and writes on a typewriter. He is terrified of how dangerous and dehumanizing technology is, and all of his books are "about" an all-consuming fear of technology and the changes it causes.

feoren 2y ago

Spiders have been a human fear for two million years, but I'm not too worried about them causing our downfall as a species. Just because we're afraid of AI doesn't mean it's actually worth fearing.

_yb2s 2y ago
Yes, even if something is worth fearing, it's not simply because people are afraid of it already. Did something I write make you think I was arguing otherwise? As an aside, I would argue that this deep rooted human fear of AI is related to the general ancient fear of humans being wiped out by other bands of more technologically advanced humans. Which for most of human history has been the single biggest danger by far. People also have deep instinctual fears of things like spiders and snakes for the same reason- they were serious hazards in the environment where humans evolved. One could say we have evolved these instincts to fear them, because they are worth fearing... not the other way around.
- feoren 2y ago
  > Did something I write make you think I was arguing otherwise? At the risk of having boring meta-commentary about comments, I feel like I see people often forgetting that their comments are in the context of a larger conversation and are going to be read as responses to the parent comment that they are, well, responding to. And then they get confused when their comments are interpreted in that context, rather than as completely standalone statements. So let's backtrack: (A) AGI is a threat to human civilization. (B) Actually, its risk is marketing hype. (You) It's not marketing hype. Our fear is deep-seated. It's not that weird to interpret the 3rd comment as disagreeing that the fear is just hype. When in fact, you were disagreeing only with the word "marketing". It's "innate" hype, not "marketing" hype. That's a much -- for lack of a better word -- weaker comment, because it is nitpicking about a minor word choice ("it's marketing hype") rather than the sentiment of the comment ("it's just hype"). So now you're asking "why did you assume my comment was interesting, and not just a little nitpick!?" > One could say we have evolved these instincts to fear them, because they are worth fearing But now you're back to saying AI is worth fearing, right? Isn't that what you're telling me I was wrong to assume you're saying? This feels like a needlessly difficult conversation. All that aside, your additions to the conversation are interesting. I actually really like your take on it being a fear of technologically superior tribes. It just doesn't feel like it needs to be quite so adversarial. You established an adversarial attitude with "it's not marketing hype".