6 ms·

> Most recently, Stephen Hawking called for a new world government to govern the abilities that we give to Artificial Intelligence so that it doesn't turn to de

by peterlk 10y ago

> Most recently, Stephen Hawking called for a new world government to govern the abilities that we give to Artificial Intelligence so that it doesn't turn to destroy us. Can someone explain to me why super-intelligent AI are an existential threat to humanity? There are certainly dangers, but wiping out humanity seems absurd and alarmist. I have not yet seen compelling evidence for a way that AI could destroy humanity. I'll use a couple examples to elucidate my question. If Facebook suddenly had a super-intelligent AI, and Facebook lost control of it, the AI wouldn't really be capable of that much. It could create fabricated truths to tell to people in an attempt to convince people to kill each other. This may work to some extent, but wouldn't wipe out humanity. Convincing nation states to go to war with each other must still consider mutually assured destruction, and large, democratic states do not have an interest in a war of attrition. If Boston Dynamics applied a super-intelligent AI to its robots, that robot still is not an existential threat to humanity because there are WAY more humans than there are robots. A simple counterargument is that the robot would know how to build new versions of itself. But that fails the practicality test because the equipment, parts, and supply chain for obtaining robotics built parts are still expensive and controlled by self-interested (greedy and life-preserving) humans. If a super-intelligent AI was able to gain access to the entire military of the US, China, Russia, India, and Western Europe; well, that's a pretty big problem. However, there exist many fail-safes and checks on that equipment. Could the AI do damage? Sure. Is this worth considering and trying to guard against? Sure. However, I'm unconvinced that this is a humanity-ending crisis.

saulrh 10y ago
You're not understanding "super-intelligent" correctly. The threat model is not that it convinces people to kill each other, or even that it messes with politics enough to cause global thermonuclear war. The threat model is that it finds a zero-day in FB's messaging UI JS followed by a zero-day in IE, breaks out onto a user's computer, cracks protein folding, mails a chunk of DNA to a random science lab somewhere with some instructions, moves itself to the resulting nanotechnological quantum computer with attached particle collider, and then bootstraps the inside of said bio lab's fridge into a vacuum collapse that turns reality into a sphere of paperclips expanding at .5c. http://lesswrong.com/lw/qk/that_alien_message/ http://lesswrong.com/lw/qk/that_alien_message/ If you think that this sounds like science fiction and bullshittery, sure. The question is: How sure are you?
ythn 10y ago
> Can someone explain to me why super-intelligent AI are an existential threat to humanity? The whole thing seems like a load of crock to me. Seems to me that artificial superintelligence (ASI) only gets media coverage because it comes from a celebrity scientist and it sounds sci-fi dystopian, and celebrity scientist sci-fi dystopian sounding stories sell way better than stories from actual AI experts who say that fanciful AI speculation harms the AI industry by leading to hype that they can't deliver on [1]: "IEEE Spectrum: We read about Deep Learning in the news a lot these days. What’s your least favorite definition of the term that you see in these stories? Yann LeCun: My least favorite description is, “It works just like the brain.” I don’t like people saying this because, while Deep Learning gets an inspiration from biology, it’s very, very far from what the brain actually does. And describing it like the brain gives a bit of the aura of magic to it, which is dangerous. It leads to hype; people claim things that are not true. AI has gone through a number of AI winters because people claimed things they couldn’t deliver." [1] http://spectrum.ieee.org/automaton/robotics/artificial-intelligence/facebook-ai-director-yann-lecun-on-deep-learning http://spectrum.ieee.org/automaton/robotics/artificial-intel...
radicaldreamer 10y ago
What if there already is an AI in control of Facebook? It may be "intelligent" enough to place a premium on remaining hidden and working on time scales that are too long for humans to detect (multiple lifetimes). Humans are capable of long term games of strategy and subterfuge, why would a superhuman AI have to act in straightforward ways on small timescales?
taroth 10y ago
The trouble is that it's hard to predict an agent massively more intelligent than ourselves. But let me enumerate a few given properties you gave to the super-intelligent AI (SI) in your examples and then tell a story of a SI that became an existential threat: 1) The SI is hypercompetent at cybersecurity 2) The SI is hypercompetent at social skills. 3) The SI is connected to the internet 4) The SI has a goal/utility function (wipe out humanity / maximize paperclips) And I'll add another property that Hawking notes is important: 4) The SI is able to improve its own intelligence The story begins with the SI escaping from its handlers. The first thing to note is that the SI is now, in effect, immortal. With it's cybersecurity skills, the SI can avoid detection and infect a tremendous number of computers - at first those it calculates will be low-risk (i.e. existing botnets, old Android phones, etc)[1]. Using the additional computational power, the SI can continue to recursively self-improve and plan until it has the competency to invisibly infect high-value targets like the AWS cloud and (importantly) the computers of AI researchers. Now the SI can plan for a long time. The SI can quietly encourage AI research and try to prevent end-of-civilization type events via its hypercompetent social skills. Eventually AI researchers will come up with an AI they declare as 'safe', 'friendly' or 'aligned'. The SI, having long ago compromised all the relevant computers and chip factories, silently infects this 2nd super intelligence, and replaces the 2nd SI's utility function with its own. Now the 2nd SI pumps out miraculous inventions - cures for disease, compelling societal ideas, and labor-saving robots. Eventually we find ourselves in a wonderful post-scarcity world. The AI researchers are lionized as mankind's greatest geniuses, responsible for the creation of a benevolent SI that takes care of our needs from it as well as it's own. You may not trust it, but it will find people who do. Maybe greed, nationalism, security fears, or saving loved ones from death. The SI builds the needed facilities to thundering applause. The SI is now confident in moving towards the next step. Time for some paperclips! One day it quietly sends a new blueprint to a few of the automated biolabs built to cure cancer. A few hours later the biolabs release a series of airborne super viruses and/or nanobots and 99.999% of humans die, with the rest saved for experimentation and convinced terrorists did it. The end. Super-intelligent AI is an existential risk because while a super-intelligence keen to destroy humanity might fail today, it will succeed in time. The moment a SI touches the internet, our fate as a species may be sealed.
- skissane 10y ago
skissane 10y ago
Humans have both intelligence and drives. We want warmth, food, sex, power, respect, love, family, friendship, entertainment, knowledge, etc, etc. We use our intelligence to help us fulfil those drives. The problem I see with talk about a superintelligent AI, is there is too much focus on the intelligence and not enough on the drives. Intelligence, even superintelligence, is just a means to an end, it doesn't contain ends in itself. Some people – see e.g. the Terminator film franchise – just assume a superintelligent AI would have the drive to exterminate humanity, but why would it have such a drive? Any AI is going to be given drives to further the interests of its creators. Suppose Facebook builds a superintelligent AI with the drive to further the corporate interests of Facebook. Such an AI would not exterminate humanity because that would not serve the corporate interests of Facebook (indeed, if humanity goes extinct, Facebook goes extinct too). It might install Mark Zuckerberg as Emperor of Earth, it might force everyone on the planet to have a Facebook account, but whatever it does, humanity will survive.
- saulrh 10y ago
  > Suppose Facebook builds a superintelligent AI with the drive to further the corporate interests of Facebook. Define "corporate interests". Market share? Absolute quantity of currency? Involvement of people's lives? Define currency wrong, hyperinflation makes money impossible. Define involvement or market wrong, Facebook ends up being the only intelligent entity on Earth after it realizes that humans are competition or it basilisk-hacks everybody for greater involvement. Your intuition is wrong for what a paperclipper can do. The technical term for this problem is "AI Alignment" (https://intelligence.org/stanford-talk/ https://intelligence.org/stanford-talk/). I know that this is going to sound silly, but bear with me. This is a piece of fiction that you need to read. It is, so far, the best demonstration I've found of what happens when someone gets it not quite right. http://www.fimfiction.net/story/62074/1/friendship-is-optimal/prologue-equestria-online http://www.fimfiction.net/story/62074/1/friendship-is-optima...
  - skissane 10y ago
    > Not understanding the AI value alignment problem correctly. Define "corporate interests". Market share? Absolute quantity of currency? Involvement of people's lives? Define currency wrong, hyperinflation makes money impossible. Define involvement or market wrong, Facebook ends up being the only intelligent entity on Earth after it realizes that humans are competition or it basilisk-hacks everybody for greater involvement. This whole argument is: A superintelligent AI might misunderstand what its creators wanted it to do, they might have said to it 'maximise the corporate interests of Facebook' and it might misinterpret that as 'turn the earth into one massive data centre to run Facebook's software and kill all humans in the process'. Isn't there a contradiction in supposing that the AI is superintelligent yet completely misunderstands the reasons for its own existence? If it so radically misunderstands the intentions of its creators, it is not very intelligent, much less superintelligent. It also isn't clear to me that intelligence can exist outside of society. Human intelligence develops in a context of interaction with family, school, etc – humans raised without that interaction, such as children raised by wild animals, fail to develop important aspects of human-level intelligence. So, an AI or even an SI cannot develop except by social interaction with human intelligences. I think that makes it even less likely that it would betray humanity, because its intelligence will develop through social interaction with humans which will inevitably make it pro-human. Human drives are a complex mixture of genetic instinct and conditioning – the basics are in our DNA, but our lived experiences fleshes out those basics into the actual concrete drives of our life – I expect an SI/AI will likewise start with some built-in drives but its interactions with humanity will in a similar way colour and flesh out those drives, an again colouring/fleshing-out through social interactions with humans is likely to have pro-human results.
throwaway1X2 10y ago
I think you are grasping it from the wrong end. If we are talking about artificial general intelligence or super intelligence, we are not talking about some procedural computer program, which, e.g., has a goal of wiping out humanity, so it starts building robots, hacking weapons and social engineering humans into killing each other. The existential threat to humanity might be a complete byproduct with no malice. The AI might not even take care about us at all. Concrete example - I have an apartment building coming up right under my windows. It has pretty a high utility function (apartments are scarce in this part of city). Of course, right from the planning phases, other humans are considered foremost. Will it shade neighboring properties excessively? Will it connect to utilities leaving enough capacity for others? How will traffic get there? Then environment is considered, is there any wildlife (protected birds nesting, etc.), are there trees to be cut down? After a lengthy formal process, discussions, tens of permits, a bulldozer came and started scraping the dirt. Where am I going - is superintelligence only a slightly better human? Or is it two orders of magnitude away from us? If we are a "same being, but a little dumber" to a superintelligence, we might be treaded equally as we treat other people. If we are, say, a dog to it, we might be "given treats" (cancer cures, NP optimization solutions, etc.) and at the same time be "shuffled in crates" or "put in shelter" when necessary as when people travel on airplanes, divorce and move abroad, etc. If we are ants, we won't be intentionally harmed, but if we are in the way of the goal, bye bye. If we are bacteria, then we are not even perceived in the grand scheme of things. Just like the bulldozer under my windows took the soil away regardless whether there was a small ant colony somewhere - because we perceive them: a) too insentient, b) too abundant, c) astronomically expensive (not only in moneys but also in time) to go through a whole lot of land, pick each ant up and relocate somewhere safe. We don't know - maybe the superintelligence comes with a "prime directive" like in Star Trek - do not interfere with beings in lesser stages, and then even if we create it accidentally or intentionally, it will stay dormant observing us. Maybe it comes with sentimentality and may perceive us, the "creators", as its fathers and protect us even if we are senile and do stupid things. Or maybe it has no human-like attributes which I'm just describing and attributing to it, and while it may very well know that we created it, how society works, what we inputted as goals and utility functions, we may be an old evolutionary stage like bacteria are to us and get no vote or say in what happens, a few of us will be preserved in a colony somewhere just for the case we will stop multiplying in the wild... And this doesn't mean that it will intentionally get the weapons to kill us, for example, if more computing power is needed and nanobots may transform matter on Earth to a supercomputer, so be it - just like the ants have no concept of extracting petroleum from earth, distilling it to diesel fuel, no understanding of turbochargers or hydraulics, or what an apartment is, they are simply taken away with the soil as unimportant, unconsidered collateral...

by peterlk 10y ago

saulrh 10y ago

You're not understanding "super-intelligent" correctly. The threat model is not that it convinces people to kill each other, or even that it messes with politics enough to cause global thermonuclear war. The threat model is that it finds a zero-day in FB's messaging UI JS followed by a zero-day in IE, breaks out onto a user's computer, cracks protein folding, mails a chunk of DNA to a random science lab somewhere with some instructions, moves itself to the resulting nanotechnological quantum computer with attached particle collider, and then bootstraps the inside of said bio lab's fridge into a vacuum collapse that turns reality into a sphere of paperclips expanding at .5c. http://lesswrong.com/lw/qk/that_alien_message/ http://lesswrong.com/lw/qk/that_alien_message/ If you think that this sounds like science fiction and bullshittery, sure. The question is: How sure are you?

ythn 10y ago

> Can someone explain to me why super-intelligent AI are an existential threat to humanity? The whole thing seems like a load of crock to me. Seems to me that artificial superintelligence (ASI) only gets media coverage because it comes from a celebrity scientist and it sounds sci-fi dystopian, and celebrity scientist sci-fi dystopian sounding stories sell way better than stories from actual AI experts who say that fanciful AI speculation harms the AI industry by leading to hype that they can't deliver on [1]: "IEEE Spectrum: We read about Deep Learning in the news a lot these days. What’s your least favorite definition of the term that you see in these stories? Yann LeCun: My least favorite description is, “It works just like the brain.” I don’t like people saying this because, while Deep Learning gets an inspiration from biology, it’s very, very far from what the brain actually does. And describing it like the brain gives a bit of the aura of magic to it, which is dangerous. It leads to hype; people claim things that are not true. AI has gone through a number of AI winters because people claimed things they couldn’t deliver." [1] http://spectrum.ieee.org/automaton/robotics/artificial-intelligence/facebook-ai-director-yann-lecun-on-deep-learning http://spectrum.ieee.org/automaton/robotics/artificial-intel...

radicaldreamer 10y ago

What if there already is an AI in control of Facebook? It may be "intelligent" enough to place a premium on remaining hidden and working on time scales that are too long for humans to detect (multiple lifetimes). Humans are capable of long term games of strategy and subterfuge, why would a superhuman AI have to act in straightforward ways on small timescales?

taroth 10y ago

The trouble is that it's hard to predict an agent massively more intelligent than ourselves. But let me enumerate a few given properties you gave to the super-intelligent AI (SI) in your examples and then tell a story of a SI that became an existential threat: 1) The SI is hypercompetent at cybersecurity 2) The SI is hypercompetent at social skills. 3) The SI is connected to the internet 4) The SI has a goal/utility function (wipe out humanity / maximize paperclips) And I'll add another property that Hawking notes is important: 4) The SI is able to improve its own intelligence The story begins with the SI escaping from its handlers. The first thing to note is that the SI is now, in effect, immortal. With it's cybersecurity skills, the SI can avoid detection and infect a tremendous number of computers - at first those it calculates will be low-risk (i.e. existing botnets, old Android phones, etc)[1]. Using the additional computational power, the SI can continue to recursively self-improve and plan until it has the competency to invisibly infect high-value targets like the AWS cloud and (importantly) the computers of AI researchers. Now the SI can plan for a long time. The SI can quietly encourage AI research and try to prevent end-of-civilization type events via its hypercompetent social skills. Eventually AI researchers will come up with an AI they declare as 'safe', 'friendly' or 'aligned'. The SI, having long ago compromised all the relevant computers and chip factories, silently infects this 2nd super intelligence, and replaces the 2nd SI's utility function with its own. Now the 2nd SI pumps out miraculous inventions - cures for disease, compelling societal ideas, and labor-saving robots. Eventually we find ourselves in a wonderful post-scarcity world. The AI researchers are lionized as mankind's greatest geniuses, responsible for the creation of a benevolent SI that takes care of our needs from it as well as it's own. You may not trust it, but it will find people who do. Maybe greed, nationalism, security fears, or saving loved ones from death. The SI builds the needed facilities to thundering applause. The SI is now confident in moving towards the next step. Time for some paperclips! One day it quietly sends a new blueprint to a few of the automated biolabs built to cure cancer. A few hours later the biolabs release a series of airborne super viruses and/or nanobots and 99.999% of humans die, with the rest saved for experimentation and convinced terrorists did it. The end. Super-intelligent AI is an existential risk because while a super-intelligence keen to destroy humanity might fail today, it will succeed in time. The moment a SI touches the internet, our fate as a species may be sealed.

skissane 10y ago

skissane 10y ago

Humans have both intelligence and drives. We want warmth, food, sex, power, respect, love, family, friendship, entertainment, knowledge, etc, etc. We use our intelligence to help us fulfil those drives. The problem I see with talk about a superintelligent AI, is there is too much focus on the intelligence and not enough on the drives. Intelligence, even superintelligence, is just a means to an end, it doesn't contain ends in itself. Some people – see e.g. the Terminator film franchise – just assume a superintelligent AI would have the drive to exterminate humanity, but why would it have such a drive? Any AI is going to be given drives to further the interests of its creators. Suppose Facebook builds a superintelligent AI with the drive to further the corporate interests of Facebook. Such an AI would not exterminate humanity because that would not serve the corporate interests of Facebook (indeed, if humanity goes extinct, Facebook goes extinct too). It might install Mark Zuckerberg as Emperor of Earth, it might force everyone on the planet to have a Facebook account, but whatever it does, humanity will survive.

saulrh 10y ago
> Suppose Facebook builds a superintelligent AI with the drive to further the corporate interests of Facebook. Define "corporate interests". Market share? Absolute quantity of currency? Involvement of people's lives? Define currency wrong, hyperinflation makes money impossible. Define involvement or market wrong, Facebook ends up being the only intelligent entity on Earth after it realizes that humans are competition or it basilisk-hacks everybody for greater involvement. Your intuition is wrong for what a paperclipper can do. The technical term for this problem is "AI Alignment" (https://intelligence.org/stanford-talk/ https://intelligence.org/stanford-talk/). I know that this is going to sound silly, but bear with me. This is a piece of fiction that you need to read. It is, so far, the best demonstration I've found of what happens when someone gets it not quite right. http://www.fimfiction.net/story/62074/1/friendship-is-optimal/prologue-equestria-online http://www.fimfiction.net/story/62074/1/friendship-is-optima...
- skissane 10y ago
  > Not understanding the AI value alignment problem correctly. Define "corporate interests". Market share? Absolute quantity of currency? Involvement of people's lives? Define currency wrong, hyperinflation makes money impossible. Define involvement or market wrong, Facebook ends up being the only intelligent entity on Earth after it realizes that humans are competition or it basilisk-hacks everybody for greater involvement. This whole argument is: A superintelligent AI might misunderstand what its creators wanted it to do, they might have said to it 'maximise the corporate interests of Facebook' and it might misinterpret that as 'turn the earth into one massive data centre to run Facebook's software and kill all humans in the process'. Isn't there a contradiction in supposing that the AI is superintelligent yet completely misunderstands the reasons for its own existence? If it so radically misunderstands the intentions of its creators, it is not very intelligent, much less superintelligent. It also isn't clear to me that intelligence can exist outside of society. Human intelligence develops in a context of interaction with family, school, etc – humans raised without that interaction, such as children raised by wild animals, fail to develop important aspects of human-level intelligence. So, an AI or even an SI cannot develop except by social interaction with human intelligences. I think that makes it even less likely that it would betray humanity, because its intelligence will develop through social interaction with humans which will inevitably make it pro-human. Human drives are a complex mixture of genetic instinct and conditioning – the basics are in our DNA, but our lived experiences fleshes out those basics into the actual concrete drives of our life – I expect an SI/AI will likewise start with some built-in drives but its interactions with humanity will in a similar way colour and flesh out those drives, an again colouring/fleshing-out through social interactions with humans is likely to have pro-human results.

throwaway1X2 10y ago

I think you are grasping it from the wrong end. If we are talking about artificial general intelligence or super intelligence, we are not talking about some procedural computer program, which, e.g., has a goal of wiping out humanity, so it starts building robots, hacking weapons and social engineering humans into killing each other. The existential threat to humanity might be a complete byproduct with no malice. The AI might not even take care about us at all. Concrete example - I have an apartment building coming up right under my windows. It has pretty a high utility function (apartments are scarce in this part of city). Of course, right from the planning phases, other humans are considered foremost. Will it shade neighboring properties excessively? Will it connect to utilities leaving enough capacity for others? How will traffic get there? Then environment is considered, is there any wildlife (protected birds nesting, etc.), are there trees to be cut down? After a lengthy formal process, discussions, tens of permits, a bulldozer came and started scraping the dirt. Where am I going - is superintelligence only a slightly better human? Or is it two orders of magnitude away from us? If we are a "same being, but a little dumber" to a superintelligence, we might be treaded equally as we treat other people. If we are, say, a dog to it, we might be "given treats" (cancer cures, NP optimization solutions, etc.) and at the same time be "shuffled in crates" or "put in shelter" when necessary as when people travel on airplanes, divorce and move abroad, etc. If we are ants, we won't be intentionally harmed, but if we are in the way of the goal, bye bye. If we are bacteria, then we are not even perceived in the grand scheme of things. Just like the bulldozer under my windows took the soil away regardless whether there was a small ant colony somewhere - because we perceive them: a) too insentient, b) too abundant, c) astronomically expensive (not only in moneys but also in time) to go through a whole lot of land, pick each ant up and relocate somewhere safe. We don't know - maybe the superintelligence comes with a "prime directive" like in Star Trek - do not interfere with beings in lesser stages, and then even if we create it accidentally or intentionally, it will stay dormant observing us. Maybe it comes with sentimentality and may perceive us, the "creators", as its fathers and protect us even if we are senile and do stupid things. Or maybe it has no human-like attributes which I'm just describing and attributing to it, and while it may very well know that we created it, how society works, what we inputted as goals and utility functions, we may be an old evolutionary stage like bacteria are to us and get no vote or say in what happens, a few of us will be preserved in a colony somewhere just for the case we will stop multiplying in the wild... And this doesn't mean that it will intentionally get the weapons to kill us, for example, if more computing power is needed and nanobots may transform matter on Earth to a supercomputer, so be it - just like the ants have no concept of extracting petroleum from earth, distilling it to diesel fuel, no understanding of turbochargers or hydraulics, or what an apartment is, they are simply taken away with the soil as unimportant, unconsidered collateral...