6 ms·

On the Catastrophic Risk of AI

25 points by bscphil 3y ago

sylware 3y ago
No. It is honestely unpredictable. And it concerns only GIGANTIC neural nets several orders of magnitude above a human brain (Telsa giga factory size, minimum).
- deleted 3y ago
  [deleted]
nico 3y ago
> I signed on to a short group statement, coordinated by the Center for AI Safety > The press coverage has been extensive, and surprising to me. The New York Times headline is “A.I. Poses ‘Risk of Extinction,’ Industry Leaders Warn.” BBC: “Artificial intelligence could lead to extinction, experts warn.” Other headlines are similar I wish we would put this level of urgency and worry into addressing human suffering in the world instead of thinking of self-serving self-aggrandizing fantastic sci-fi futures that are just distracting us from the current reality of society A lot more people suffer and will suffer everyday from inequality, health issues and loneliness than anything that any AI will do to them in their lives
andrewstuart 3y ago
So far I haven’t read anything that convinces me of this catastrophic risk from AI. All I’ve read is science fiction from man-boys speculating on some sort of fanciful future that isn’t even close to current reality. People like Sam Altman catastrophizing to politicians and panicking people about “Tom Swift and the electric brain” is frankly doing the industry a disservice and embarrassing. AI ending humanity is on the same likelihood level as man-boy Musks dreams of colonizing Mars…. I know people love the fantasy, but it will never be real. People should just stop with this made up stuff or the politicians will believe you and their stupidity is the real risk. Climate change is a million times more real existential threat.
- famouswaffles 3y ago
  Unaligned intelligence is dangerous. You don't need a Sci-fi book to realize this, just a history one. Our history is replete with examples of the slightly more technologically advanced group decimating their competition. Humanity is directly responsible for the extinction of hundreds of species. Not because of any particular malice or offense, simply that our goals didn't align with the interest of any of those species. Just looking at Humans alone, Unaligned intelligence is potentially catastrophic even when the advantage gulf is fairly low. When the advantage gulf is large, it is potentially genocidal.
  - andrewstuart 3y ago
    Still these are vague sentiments.
    - NumberWangMan 3y ago
      I've found a sort of pattern in people's reluctance to believe that AGI poses an existential risk. If you think of specific scenarios about how AIs could take over the world or kill everyone, people will say, "oh, that's easy to avoid, we can just do X". If you think of scenarios that are hard to avoid, like an AGI designing and releasing a deadly, virulent virus (or lots of different ones), people will say, "but why would AGI want to kill everyone?" If you point out how AGI might have very good reasons to get rid of humans, and that we don't have any way of really guaranteeing they won't try, people will say, "but we won't give them any power". If you point out that once we make AGI, it's inevitable that somebody somewhere will give it power, and even if they don't, it's very easy for someone very intelligent and sociopathic to manipulate gullible people, they say something like, "well, we superintelligence probably isn't even possible", or "we can't even agree on what intelligence means". Turn it around for a second. What does it take for humanity to have a good future after we create intelligences smarter than us? Let's say the AGI listens to and obeys humans. Well, one thing we need to avoid is just an absolute chaos of terrorism and war. It's easier to do damage than to prevent it, and if everyone has an obedient, mastermind scientific genius in their back pocket, you could have it invent a pathogen and cure in tandem, keep the cure secret and release the pathogen, and do this over and over until your enemies are all dead. So we need AGI that is aligned with humanity, in the sense that it won't do stuff that hurts people even if someone tells it to. How do we do that, and not potentially end up with AGIs that can't be corrected if they get some idea that turns out to be wrong? Ideally, we'd make AGIs that have a bias toward inaction, that are "lazy" in a sense, and try to do the minimum they can. But those AGIs won't be as useful as ones that work like eager beavers toward some goal, so there will be selection pressure towards those. I think the way to look at it is like with evolutionary biology. The pressure that evolution puts on animals to develop certain genes/behaviors is similar to the pressure that humans will put on AI development, and it ends with AI being more capable and more likely to follow orders even if those orders mean bad things for other people. As they get more and more powerful, this means chaos. It's monkeys throwing poo, vs monkeys throwing rocks, vs monkeys with guns, vs monkeys with rocket launchers. And look -- I'm not certain that things will end in disaster. But inventing something smarter than you and trying to stop it from taking over sooner or later seems very difficult, and I can't even get a good answer out of people who think it'll be fine, or good. The default case is not that it'll be fine, the default case is we have no idea how it'll go. It's a grey fog. Just because humanity has muddled along and managed not to destroy ourselves so far, does not mean that we're safe, by any means.
tester457 3y ago
> we didn't build ChatGPT, in the sense most people would use the word. We built a training process that spat out ChatGPT. Nobody can actually build ChatGPT, and nobody really knows how or why it does the things it does. - Rob Miles In the future this might be how we create AGI, without a deeper understanding of it. This will be worse than any kernel bug because at least in OS security no one dies if a bug is found and you have multiple tries to bug fix your OS. AGI will be guaranteed to have bugs, like everything does.
- ramijames 3y ago
  People have bugs. Lots of them.
  - loonginthetooth 3y ago
    I think this is kind of the point. There's a compelling argument that buggy humans are the root of all kinds of profound suffering.
- ilaksh 3y ago
  What does AGI mean to you? Something that is an exact simulation of a person, but living inside a computer?
  - tester457 3y ago
    Something that can solve problems better than the average human. It doesn't have to understand the problem either or be conscious. Solving it is enough.
Imnimo 3y ago
Even at only one sentence long, people will apparently sign these things because they like the vibes of the statement, despite disagreeing with the actual language.
mola 3y ago
So he doesn't think AI is a risk for extinction. But he signed off on it. Doesn't he understand he was duped. Why was he surprised by the coverage? Does he not understand that this whole thing is a PR stunt by OpenAI for Mr Altman's panic circus? My god, the nativity....
LatteLazy 3y ago
>Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war. So we should do nothing at all then? And actively make the risks worse if it serves narrow national, political interests?
ilaksh 3y ago
We have to think of nuclear war and pandemics as extinction risks even though we probably can find ways to avoid completely dying out, because if we really fail we can still die out. Similar thing with AI. But actually probably more dangerous. Maybe Schneier isn't as smart as everyone thinks he is. GPT-4 is already almost human level intelligence. (Notice that I did not say that it was alive or that it was a simulation of a person with all of the other human characteristics). All it takes is for them to keep making computers faster and more efficient like they have been since it started. Within five years or less it will be thinking 100 times faster than humans (and quite possibly double the IQ). Then someone people gives a large group of them an overly broad instruction like "take control". GPT-4 is truly an effective problem solver and similar systems will be available within a few years that are orders of magnitude faster. That is an extinction risk. You cannot compete with it as a human. For this type of system, the human activities or inputs are so slow that they seem to be frozen. People may think that computing progress has ended or slowed down or something and it can't get that much faster, but for starters this is a very specific application which has room for optimization in hardware, software, and AI models. And there are entirely new compute-in-memory paradigms coming (a new S curves).

25 points by bscphil 3y ago

No. It is honestely unpredictable. And it concerns only GIGANTIC neural nets several orders of magnitude above a human brain (Telsa giga factory size, minimum).

deleted 3y ago
[deleted]

nico 3y ago

> I signed on to a short group statement, coordinated by the Center for AI Safety > The press coverage has been extensive, and surprising to me. The New York Times headline is “A.I. Poses ‘Risk of Extinction,’ Industry Leaders Warn.” BBC: “Artificial intelligence could lead to extinction, experts warn.” Other headlines are similar I wish we would put this level of urgency and worry into addressing human suffering in the world instead of thinking of self-serving self-aggrandizing fantastic sci-fi futures that are just distracting us from the current reality of society A lot more people suffer and will suffer everyday from inequality, health issues and loneliness than anything that any AI will do to them in their lives

andrewstuart 3y ago

So far I haven’t read anything that convinces me of this catastrophic risk from AI. All I’ve read is science fiction from man-boys speculating on some sort of fanciful future that isn’t even close to current reality. People like Sam Altman catastrophizing to politicians and panicking people about “Tom Swift and the electric brain” is frankly doing the industry a disservice and embarrassing. AI ending humanity is on the same likelihood level as man-boy Musks dreams of colonizing Mars…. I know people love the fantasy, but it will never be real. People should just stop with this made up stuff or the politicians will believe you and their stupidity is the real risk. Climate change is a million times more real existential threat.

famouswaffles 3y ago
Unaligned intelligence is dangerous. You don't need a Sci-fi book to realize this, just a history one. Our history is replete with examples of the slightly more technologically advanced group decimating their competition. Humanity is directly responsible for the extinction of hundreds of species. Not because of any particular malice or offense, simply that our goals didn't align with the interest of any of those species. Just looking at Humans alone, Unaligned intelligence is potentially catastrophic even when the advantage gulf is fairly low. When the advantage gulf is large, it is potentially genocidal.
- andrewstuart 3y ago
  Still these are vague sentiments.
  - NumberWangMan 3y ago
    I've found a sort of pattern in people's reluctance to believe that AGI poses an existential risk. If you think of specific scenarios about how AIs could take over the world or kill everyone, people will say, "oh, that's easy to avoid, we can just do X". If you think of scenarios that are hard to avoid, like an AGI designing and releasing a deadly, virulent virus (or lots of different ones), people will say, "but why would AGI want to kill everyone?" If you point out how AGI might have very good reasons to get rid of humans, and that we don't have any way of really guaranteeing they won't try, people will say, "but we won't give them any power". If you point out that once we make AGI, it's inevitable that somebody somewhere will give it power, and even if they don't, it's very easy for someone very intelligent and sociopathic to manipulate gullible people, they say something like, "well, we superintelligence probably isn't even possible", or "we can't even agree on what intelligence means". Turn it around for a second. What does it take for humanity to have a good future after we create intelligences smarter than us? Let's say the AGI listens to and obeys humans. Well, one thing we need to avoid is just an absolute chaos of terrorism and war. It's easier to do damage than to prevent it, and if everyone has an obedient, mastermind scientific genius in their back pocket, you could have it invent a pathogen and cure in tandem, keep the cure secret and release the pathogen, and do this over and over until your enemies are all dead. So we need AGI that is aligned with humanity, in the sense that it won't do stuff that hurts people even if someone tells it to. How do we do that, and not potentially end up with AGIs that can't be corrected if they get some idea that turns out to be wrong? Ideally, we'd make AGIs that have a bias toward inaction, that are "lazy" in a sense, and try to do the minimum they can. But those AGIs won't be as useful as ones that work like eager beavers toward some goal, so there will be selection pressure towards those. I think the way to look at it is like with evolutionary biology. The pressure that evolution puts on animals to develop certain genes/behaviors is similar to the pressure that humans will put on AI development, and it ends with AI being more capable and more likely to follow orders even if those orders mean bad things for other people. As they get more and more powerful, this means chaos. It's monkeys throwing poo, vs monkeys throwing rocks, vs monkeys with guns, vs monkeys with rocket launchers. And look -- I'm not certain that things will end in disaster. But inventing something smarter than you and trying to stop it from taking over sooner or later seems very difficult, and I can't even get a good answer out of people who think it'll be fine, or good. The default case is not that it'll be fine, the default case is we have no idea how it'll go. It's a grey fog. Just because humanity has muddled along and managed not to destroy ourselves so far, does not mean that we're safe, by any means.

tester457 3y ago

> we didn't build ChatGPT, in the sense most people would use the word. We built a training process that spat out ChatGPT. Nobody can actually build ChatGPT, and nobody really knows how or why it does the things it does. - Rob Miles In the future this might be how we create AGI, without a deeper understanding of it. This will be worse than any kernel bug because at least in OS security no one dies if a bug is found and you have multiple tries to bug fix your OS. AGI will be guaranteed to have bugs, like everything does.

ramijames 3y ago
People have bugs. Lots of them.
- loonginthetooth 3y ago
  I think this is kind of the point. There's a compelling argument that buggy humans are the root of all kinds of profound suffering.
ilaksh 3y ago
What does AGI mean to you? Something that is an exact simulation of a person, but living inside a computer?
- tester457 3y ago
  Something that can solve problems better than the average human. It doesn't have to understand the problem either or be conscious. Solving it is enough.

Imnimo 3y ago

Even at only one sentence long, people will apparently sign these things because they like the vibes of the statement, despite disagreeing with the actual language.

mola 3y ago

So he doesn't think AI is a risk for extinction. But he signed off on it. Doesn't he understand he was duped. Why was he surprised by the coverage? Does he not understand that this whole thing is a PR stunt by OpenAI for Mr Altman's panic circus? My god, the nativity....

LatteLazy 3y ago

>Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war. So we should do nothing at all then? And actively make the risks worse if it serves narrow national, political interests?

ilaksh 3y ago

We have to think of nuclear war and pandemics as extinction risks even though we probably can find ways to avoid completely dying out, because if we really fail we can still die out. Similar thing with AI. But actually probably more dangerous. Maybe Schneier isn't as smart as everyone thinks he is. GPT-4 is already almost human level intelligence. (Notice that I did not say that it was alive or that it was a simulation of a person with all of the other human characteristics). All it takes is for them to keep making computers faster and more efficient like they have been since it started. Within five years or less it will be thinking 100 times faster than humans (and quite possibly double the IQ). Then someone people gives a large group of them an overly broad instruction like "take control". GPT-4 is truly an effective problem solver and similar systems will be available within a few years that are orders of magnitude faster. That is an extinction risk. You cannot compete with it as a human. For this type of system, the human activities or inputs are so slow that they seem to be frozen. People may think that computing progress has ended or slowed down or something and it can't get that much faster, but for starters this is a very specific application which has room for optimization in hardware, software, and AI models. And there are entirely new compute-in-memory paradigms coming (a new S curves).