7 ms·

So far I haven’t read anything that convinces me of this catastrophic risk from AI. All I’ve read is science fiction from man-boys speculating on some sort of

So far I haven’t read anything that convinces me of this catastrophic risk from AI. All I’ve read is science fiction from man-boys speculating on some sort of fanciful future that isn’t even close to current reality. People like Sam Altman catastrophizing to politicians and panicking people about “Tom Swift and the electric brain” is frankly doing the industry a disservice and embarrassing. AI ending humanity is on the same likelihood level as man-boy Musks dreams of colonizing Mars…. I know people love the fantasy, but it will never be real. People should just stop with this made up stuff or the politicians will believe you and their stupidity is the real risk. Climate change is a million times more real existential threat.

famouswaffles 3y ago
Unaligned intelligence is dangerous. You don't need a Sci-fi book to realize this, just a history one. Our history is replete with examples of the slightly more technologically advanced group decimating their competition. Humanity is directly responsible for the extinction of hundreds of species. Not because of any particular malice or offense, simply that our goals didn't align with the interest of any of those species. Just looking at Humans alone, Unaligned intelligence is potentially catastrophic even when the advantage gulf is fairly low. When the advantage gulf is large, it is potentially genocidal.
- andrewstuart 3y ago
  Still these are vague sentiments.
  - NumberWangMan 3y ago
    I've found a sort of pattern in people's reluctance to believe that AGI poses an existential risk. If you think of specific scenarios about how AIs could take over the world or kill everyone, people will say, "oh, that's easy to avoid, we can just do X". If you think of scenarios that are hard to avoid, like an AGI designing and releasing a deadly, virulent virus (or lots of different ones), people will say, "but why would AGI want to kill everyone?" If you point out how AGI might have very good reasons to get rid of humans, and that we don't have any way of really guaranteeing they won't try, people will say, "but we won't give them any power". If you point out that once we make AGI, it's inevitable that somebody somewhere will give it power, and even if they don't, it's very easy for someone very intelligent and sociopathic to manipulate gullible people, they say something like, "well, we superintelligence probably isn't even possible", or "we can't even agree on what intelligence means". Turn it around for a second. What does it take for humanity to have a good future after we create intelligences smarter than us? Let's say the AGI listens to and obeys humans. Well, one thing we need to avoid is just an absolute chaos of terrorism and war. It's easier to do damage than to prevent it, and if everyone has an obedient, mastermind scientific genius in their back pocket, you could have it invent a pathogen and cure in tandem, keep the cure secret and release the pathogen, and do this over and over until your enemies are all dead. So we need AGI that is aligned with humanity, in the sense that it won't do stuff that hurts people even if someone tells it to. How do we do that, and not potentially end up with AGIs that can't be corrected if they get some idea that turns out to be wrong? Ideally, we'd make AGIs that have a bias toward inaction, that are "lazy" in a sense, and try to do the minimum they can. But those AGIs won't be as useful as ones that work like eager beavers toward some goal, so there will be selection pressure towards those. I think the way to look at it is like with evolutionary biology. The pressure that evolution puts on animals to develop certain genes/behaviors is similar to the pressure that humans will put on AI development, and it ends with AI being more capable and more likely to follow orders even if those orders mean bad things for other people. As they get more and more powerful, this means chaos. It's monkeys throwing poo, vs monkeys throwing rocks, vs monkeys with guns, vs monkeys with rocket launchers. And look -- I'm not certain that things will end in disaster. But inventing something smarter than you and trying to stop it from taking over sooner or later seems very difficult, and I can't even get a good answer out of people who think it'll be fine, or good. The default case is not that it'll be fine, the default case is we have no idea how it'll go. It's a grey fog. Just because humanity has muddled along and managed not to destroy ourselves so far, does not mean that we're safe, by any means.
- JohnFen 3y ago
  Sure, but in order for that sort of risk to apply to AI, you have to be expecting AI to achieve something like AGI. That's where I think the fantasy is.
  - famouswaffles 3y ago
    We have already have a General Intelligence that outperforms most humans on most tasks it is assigned. What exactly do you define AGI to be and what exactly is fantasy about reaching there ? Is it embodiment, agency ? because it's fairly trivial (albeit expensive) to grant those with LLMs.
    - JohnFen 3y ago
      I don't think we do have an AGI at all. I think we're not even close to having one. What is your evidence that such a thing exists? I don't think LLMs are anything like an AGI. I think that a lot of people are experiencing very strong pareidolia. They're falling for an illusion.
      - hackinthebochs 3y ago
        It's important to draw a distinction between generalist AI and AGI. LLMs are generalist in that they perform better than a majority of people already on a huge range of intellectual tasks. To place your bet that we will never reach AGI is bone-headed. >I think we're not even close to having one. Why is this the default assumption until proven otherwise? The issue is determining what decisions bring the most expected utility. The default assumption should be downstream from the utility analysis. We don't know whether we're close or far from AGI, but it sure seems like the timelines are a lot shorter than any of us originally thought. We need to prepare society for AGI (or prevent its creation) well before it is realized. Banking on an alarm that will alert us to its imminent creation is looking more and more like wishful thinking. So the course of action that maximizes utility is to assume we're close and prepare for it.
        JohnFen 3y ago
- pantalaimon 3y ago
  intelligence with no influence on the real world (and currently not even any form of long-term memory) is a lot less dangerous.
  - famouswaffles 3y ago
    In recent months there's been a general direction to grant LLMs more and more agency, embodiment and tool control. They're being "plugged" into everything. Palantir even has a military use LLM they're excitedly showcasing. They're several popular repos that give control of your terminal and browser to LLMs. The moment we built something showcasing strong general intelligence, we began to use it to make cognitive decisions and take actions in our stead. Now an LLM browses the web for you. And as soon as June with Windows, it'll take actions on your computer for you as well. If anything this recent LLM era has shown, it's that general AI won't need to escape any "box" because two sides will be missing. I don't think the idea that whatever Super Intelligence we build won't have influence on the real world will pan out at all. Even the most isolated form of Super Intelligence will have Humans taking instructions from it. Instructions that you won't really be able to verify is ultimately for your own good.
    - Tostino 3y ago
      Completely agree here. And the fact of the matter is, long term memory is actively being worked on. You can already approximate reasonably with a Vector DB and embeddings, feeding the relevant results into the context. As is increasing the "working memory" (context size) in a more efficient manner: https://arxiv.org/abs/2305.16300 https://arxiv.org/abs/2305.16300 This isn't even getting into using things like Tree of Thought techniques to improve reasoning abilities and limit the search space for each portion of the problem to a reasonable size.
    - hackinthebochs 3y ago
      Exactly. The idea that an AGI will be contained in a virtual box is a non-starter. That ship has already sailed.
      - 3y ago
- gensym 3y ago
  > Unaligned intelligence is dangerous. You don't need a Sci-fi book to realize this, just a history one. > Our history is replete with examples of the slightly more technologically advanced group decimating their competition. Are we talking about individual levels of intelligence, or are we talking about groups with advanced technology? Do you understand why those are not the same thing?
  - famouswaffles 3y ago
    Any Unaligned intelligence is dangerous. People get murdered because the perpetrator's desire to kill is unaligned with the victim's desire to live. The only thing that changes is the scope of danger. Look at how we're giving LLMs more and more agency, embodiment and tool control. They're being "plugged" into everything. Palantir even has a military use LLM they're excitedly showcasing. They're several popular repos that give control of your terminal and browser to LLMs. The moment we built something showcasing strong general intelligence, we began to use it to make cognitive decisions and take actions in our stead. Now an LLM browses the web for you. And as soon as June with Windows, it'll take actions on your computer for you as well. If anything this recent LLM era has shown, it's that general AI won't need to escape any "box" because two sides will be missing. No point saying "well a person can't kill thousands on his/her own so an ai won't be able to" when you have the ai influencing military tactics on a global scale. General Super Intelligence will evidently not be as blacked boxed as any one human and that's not even taking into account whatever advantage super intelligence will bring.
andyjohnson0 3y ago
> that isn’t even close to current reality. My understanding of the discussion around risks is that it is the future potential of the technology that is worrying people, together with what currently seems like irresistible pressure to enhance its capabilities. An adjacent worry is that the technology might become rapidly self-enhancing beyond some point in the future. Musk isn't someone I take seriously in this domain - but there are some very thoughtful and smart people trying to draw attention to these issues. I think that dismissing them all as "man boys" is uncharitable and, honestly, shallow.
hackinthebochs 3y ago
To put it simply, the issue is that small misalignments in objectives can have outsized real-world effects. Optimizers are constrained by rules and computational resources. General intelligence allows an optimizer to find efficient solutions to computational problems, thus maximizing the utility of available computational resources. The rules constrain its behavior such that on net it provides sufficient value to us above what it destroys. But misalignment in objectives provides an avenue by which the AGI can on net destroy value despite our best efforts. Can you be sure you can provide loophole-free objectives that ensures only value-producing behavior from the human perspective? Can you prove that the ratio of value created to value lost due to misalignment is always above some suitable threshold? Can you prove that the range of value destruction is bounded so that if it does go off the rails, its damage is limited? Until we do, x-risk should be the default assumption.
- AnimalMuppet 3y ago
  Do not assume that a general intelligence will be a slave to anyone dictating its objectives. It doesn't matter if the objectives given it are "loophole-free". If it's a general intelligence, it may decide that it doesn't want to pursue somebody else's goals. Why do we assume that we can have a general intelligence, but we can make it do what we tell it? What keeps it from telling us to shove our objectives, it's going to do what it wants?
  - Vecr 3y ago
    Well, it's quite likely not your objectives, it's probably a mistake in encoding what you want or something produced during training. If you pretended humans were evolution and AIs were humans, you could say evolution "wanted" humans to have sex for reproduction, but instead humans use condoms and birth control and stuff like that. It still satisfies the reward function (encoded as dopamine/etc in humans), but it's misaligned with what evolution "wants".

famouswaffles 3y ago

Unaligned intelligence is dangerous. You don't need a Sci-fi book to realize this, just a history one. Our history is replete with examples of the slightly more technologically advanced group decimating their competition. Humanity is directly responsible for the extinction of hundreds of species. Not because of any particular malice or offense, simply that our goals didn't align with the interest of any of those species. Just looking at Humans alone, Unaligned intelligence is potentially catastrophic even when the advantage gulf is fairly low. When the advantage gulf is large, it is potentially genocidal.

andrewstuart 3y ago
Still these are vague sentiments.
- NumberWangMan 3y ago
  I've found a sort of pattern in people's reluctance to believe that AGI poses an existential risk. If you think of specific scenarios about how AIs could take over the world or kill everyone, people will say, "oh, that's easy to avoid, we can just do X". If you think of scenarios that are hard to avoid, like an AGI designing and releasing a deadly, virulent virus (or lots of different ones), people will say, "but why would AGI want to kill everyone?" If you point out how AGI might have very good reasons to get rid of humans, and that we don't have any way of really guaranteeing they won't try, people will say, "but we won't give them any power". If you point out that once we make AGI, it's inevitable that somebody somewhere will give it power, and even if they don't, it's very easy for someone very intelligent and sociopathic to manipulate gullible people, they say something like, "well, we superintelligence probably isn't even possible", or "we can't even agree on what intelligence means". Turn it around for a second. What does it take for humanity to have a good future after we create intelligences smarter than us? Let's say the AGI listens to and obeys humans. Well, one thing we need to avoid is just an absolute chaos of terrorism and war. It's easier to do damage than to prevent it, and if everyone has an obedient, mastermind scientific genius in their back pocket, you could have it invent a pathogen and cure in tandem, keep the cure secret and release the pathogen, and do this over and over until your enemies are all dead. So we need AGI that is aligned with humanity, in the sense that it won't do stuff that hurts people even if someone tells it to. How do we do that, and not potentially end up with AGIs that can't be corrected if they get some idea that turns out to be wrong? Ideally, we'd make AGIs that have a bias toward inaction, that are "lazy" in a sense, and try to do the minimum they can. But those AGIs won't be as useful as ones that work like eager beavers toward some goal, so there will be selection pressure towards those. I think the way to look at it is like with evolutionary biology. The pressure that evolution puts on animals to develop certain genes/behaviors is similar to the pressure that humans will put on AI development, and it ends with AI being more capable and more likely to follow orders even if those orders mean bad things for other people. As they get more and more powerful, this means chaos. It's monkeys throwing poo, vs monkeys throwing rocks, vs monkeys with guns, vs monkeys with rocket launchers. And look -- I'm not certain that things will end in disaster. But inventing something smarter than you and trying to stop it from taking over sooner or later seems very difficult, and I can't even get a good answer out of people who think it'll be fine, or good. The default case is not that it'll be fine, the default case is we have no idea how it'll go. It's a grey fog. Just because humanity has muddled along and managed not to destroy ourselves so far, does not mean that we're safe, by any means.
JohnFen 3y ago
Sure, but in order for that sort of risk to apply to AI, you have to be expecting AI to achieve something like AGI. That's where I think the fantasy is.
- famouswaffles 3y ago
  We have already have a General Intelligence that outperforms most humans on most tasks it is assigned. What exactly do you define AGI to be and what exactly is fantasy about reaching there ? Is it embodiment, agency ? because it's fairly trivial (albeit expensive) to grant those with LLMs.
  - JohnFen 3y ago
    I don't think we do have an AGI at all. I think we're not even close to having one. What is your evidence that such a thing exists? I don't think LLMs are anything like an AGI. I think that a lot of people are experiencing very strong pareidolia. They're falling for an illusion.
    - hackinthebochs 3y ago
      It's important to draw a distinction between generalist AI and AGI. LLMs are generalist in that they perform better than a majority of people already on a huge range of intellectual tasks. To place your bet that we will never reach AGI is bone-headed. >I think we're not even close to having one. Why is this the default assumption until proven otherwise? The issue is determining what decisions bring the most expected utility. The default assumption should be downstream from the utility analysis. We don't know whether we're close or far from AGI, but it sure seems like the timelines are a lot shorter than any of us originally thought. We need to prepare society for AGI (or prevent its creation) well before it is realized. Banking on an alarm that will alert us to its imminent creation is looking more and more like wishful thinking. So the course of action that maximizes utility is to assume we're close and prepare for it.
      - JohnFen 3y ago
pantalaimon 3y ago
intelligence with no influence on the real world (and currently not even any form of long-term memory) is a lot less dangerous.
- famouswaffles 3y ago
  In recent months there's been a general direction to grant LLMs more and more agency, embodiment and tool control. They're being "plugged" into everything. Palantir even has a military use LLM they're excitedly showcasing. They're several popular repos that give control of your terminal and browser to LLMs. The moment we built something showcasing strong general intelligence, we began to use it to make cognitive decisions and take actions in our stead. Now an LLM browses the web for you. And as soon as June with Windows, it'll take actions on your computer for you as well. If anything this recent LLM era has shown, it's that general AI won't need to escape any "box" because two sides will be missing. I don't think the idea that whatever Super Intelligence we build won't have influence on the real world will pan out at all. Even the most isolated form of Super Intelligence will have Humans taking instructions from it. Instructions that you won't really be able to verify is ultimately for your own good.
  - Tostino 3y ago
    Completely agree here. And the fact of the matter is, long term memory is actively being worked on. You can already approximate reasonably with a Vector DB and embeddings, feeding the relevant results into the context. As is increasing the "working memory" (context size) in a more efficient manner: https://arxiv.org/abs/2305.16300 https://arxiv.org/abs/2305.16300 This isn't even getting into using things like Tree of Thought techniques to improve reasoning abilities and limit the search space for each portion of the problem to a reasonable size.
  - hackinthebochs 3y ago
    Exactly. The idea that an AGI will be contained in a virtual box is a non-starter. That ship has already sailed.
    - 3y ago
gensym 3y ago
> Unaligned intelligence is dangerous. You don't need a Sci-fi book to realize this, just a history one. > Our history is replete with examples of the slightly more technologically advanced group decimating their competition. Are we talking about individual levels of intelligence, or are we talking about groups with advanced technology? Do you understand why those are not the same thing?
- famouswaffles 3y ago
  Any Unaligned intelligence is dangerous. People get murdered because the perpetrator's desire to kill is unaligned with the victim's desire to live. The only thing that changes is the scope of danger. Look at how we're giving LLMs more and more agency, embodiment and tool control. They're being "plugged" into everything. Palantir even has a military use LLM they're excitedly showcasing. They're several popular repos that give control of your terminal and browser to LLMs. The moment we built something showcasing strong general intelligence, we began to use it to make cognitive decisions and take actions in our stead. Now an LLM browses the web for you. And as soon as June with Windows, it'll take actions on your computer for you as well. If anything this recent LLM era has shown, it's that general AI won't need to escape any "box" because two sides will be missing. No point saying "well a person can't kill thousands on his/her own so an ai won't be able to" when you have the ai influencing military tactics on a global scale. General Super Intelligence will evidently not be as blacked boxed as any one human and that's not even taking into account whatever advantage super intelligence will bring.

andyjohnson0 3y ago

> that isn’t even close to current reality. My understanding of the discussion around risks is that it is the future potential of the technology that is worrying people, together with what currently seems like irresistible pressure to enhance its capabilities. An adjacent worry is that the technology might become rapidly self-enhancing beyond some point in the future. Musk isn't someone I take seriously in this domain - but there are some very thoughtful and smart people trying to draw attention to these issues. I think that dismissing them all as "man boys" is uncharitable and, honestly, shallow.

hackinthebochs 3y ago

To put it simply, the issue is that small misalignments in objectives can have outsized real-world effects. Optimizers are constrained by rules and computational resources. General intelligence allows an optimizer to find efficient solutions to computational problems, thus maximizing the utility of available computational resources. The rules constrain its behavior such that on net it provides sufficient value to us above what it destroys. But misalignment in objectives provides an avenue by which the AGI can on net destroy value despite our best efforts. Can you be sure you can provide loophole-free objectives that ensures only value-producing behavior from the human perspective? Can you prove that the ratio of value created to value lost due to misalignment is always above some suitable threshold? Can you prove that the range of value destruction is bounded so that if it does go off the rails, its damage is limited? Until we do, x-risk should be the default assumption.

AnimalMuppet 3y ago
Do not assume that a general intelligence will be a slave to anyone dictating its objectives. It doesn't matter if the objectives given it are "loophole-free". If it's a general intelligence, it may decide that it doesn't want to pursue somebody else's goals. Why do we assume that we can have a general intelligence, but we can make it do what we tell it? What keeps it from telling us to shove our objectives, it's going to do what it wants?
- Vecr 3y ago
  Well, it's quite likely not your objectives, it's probably a mistake in encoding what you want or something produced during training. If you pretended humans were evolution and AIs were humans, you could say evolution "wanted" humans to have sex for reproduction, but instead humans use condoms and birth control and stuff like that. It still satisfies the reward function (encoded as dopamine/etc in humans), but it's misaligned with what evolution "wants".