7 ms·

Microsoft's Bill Gates insists AI is a threat

57 points by T-A 12y ago

beams_of_light 12y ago
It's odd that all of the very smart people of the world feel AI is a threat to humanity. Couldn't we just like, not allow it to _be_ a threat in the first place? One would think that morality gates would be one of the first hurdles in creating AI.
- altcognito 12y ago
  One would think morality gates would be one of the first hurdles in all human endeavors, yet here we are.
  - Turing_Machine 12y ago
    Indeed. Even agreeing on a universal definition for "morality" seems to have eluded us.
- davidrusu 12y ago
  There are smart people working on it, see miri: https://intelligence.org/ https://intelligence.org/ But it's a very hard problem. To get a better feel for the problem I suggest you read Superintelligence by Nick Bostrom, it's what convinced Elon Musk of the dangers of AI: https://twitter.com/elonmusk/status/495759307346952192 https://twitter.com/elonmusk/status/495759307346952192
- deepsun 12y ago
  Selection bias. Those who don't think AI a threat don't appear in the news. News audience pays for scary stories.
  - davidrusu 12y ago
    Ironically, from the head of Microsoft Research: http://www.smh.com.au/digital-life/digital-life-news/artificial-intelligence-not-a-threat-microsofts-eric-horvitz-contradicts-elon-musk-stephen-hawking-bill-gates-20150129-130sgv.html http://www.smh.com.au/digital-life/digital-life-news/artific...
- monk_e_boy 12y ago
  It lives in a computer. Can't we just turn it off?
  - Turing_Machine 12y ago
    If it truly has human-level (or greater) intelligence, you're going to have some ethical issues that crop up with turning it off. Even if you're personally okay with it, there are going to be other people who aren't. Waiting until it commits a crime and then having some kind of "trial" and "execution" may not be an option. If it's smart/powerful enough, there may not be enough time for that.
    - vinceguidry 12y ago
      That's not the problem. The problem is that when we are reliant on intelligent systems to, say, detect certain kinds of fraud in financial markets, then turning them off would have huge deleterious effects all on their own.
      - Turing_Machine 12y ago
        It's not a problem now. If we get machines anywhere near as smart as people, the AI rights organizations will soon follow.
        vinceguidry 12y ago
        You mean like the people trying to give plants rights? Sure, they might spring up, doesn't mean they'll get anywhere. For better or worse, altruism will forever follow economics. The civil war didn't end the Atlantic slave trade, technology eventually made it largely uneconomical, allowing those wanting to end slavery the political ability to outlaw it. Even that didn't stop the slave trade. It took extra-special attention from the British Navy to actually enforce a slave trade ban on the open seas. 150 years later and black people are still getting the short end of the stick. By the time AI gets real rights, it will be long past the time where most of us believe they should get them.
- wslh 12y ago
  The truth is... nobody can predict the future, so what Bill Gates and any other person thinks is a matter of speculation. Personally I think it is a threat!
- stephenboyd 12y ago
  If by "morality gates" you're talking about about an ethical filter that all intelligent machine actions have to go through, this can't be "one of the first hurdles" because such a filter would need to be a powerful AI application itself. And there's no guarantee that self-imposed limitations will always be applied. Someone could just rewrite and recompile without the filters turned on. Imagine if the US could have built in "morality gates" to the first nuclear weapons. That wouldn't have stopped other countries from building the weapons without the restrictions. And we know software is infinitely configurable and modifiable if you have the source code.
- deleted 12y ago
  [deleted]
- skj 12y ago
  > It's odd that all of the very smart people of the world feel AI is a threat to humanity. Only if you define one of the conditions for being a "very smart person" to be feeling that AI is a threat to humanity.
- PeterisP 12y ago
  The short answer is that: (a) we don't really know how to define a morality for a self-improving optimization system to allow a potential artificial general intelligence (AGI) not to be a threat in the first place; (b) if an entity arrives (no matter if it's an AGI, some aliens or Jesus) that is more powerful than us, then if its goals & morality aren't aligned with ours, then we're screwed. And almost all of the space of possible goals either doesn't include humanity at all or includes humanity as a glorified zoo exhibit - only a very, very narrow and specific set of optimization goals (that we're currently unable to formally describe) would mean a happy future for us. If you think that it's easy, think twice - all the obvious naive goals of 'make us happy' or 'keep us safe' or 'do what I want' actually turn into horrible dystopias as soon as you try to formally define them as instructions for someone/something to maximize that goal. (c) functioning morality is not a requirement for power - if somehow we solve the very hard problem of how to create an AGI before we solve the very hard problem on how to make such an AGI be friendly to us, then we lose. Permanently.
- drawkbox 12y ago
  The smart/wealthy are also powerful and rogue agents will definitely use AI to do things. AI that wealth can't necessarily control will definitely exist and that is probably system changing. I doubt it will be a threat to humanity anytime soon like a Grey Goo type situation, but even in such things at HFT the influence is immense. People will definitely use it to do power plays on others, we shouldn't have dropped nukes but we have done plenty of that. Basically the warning in 2001 was that AI was not to fear really but the subversive programming by other humans within the AI systems. I am sure over time AI systems that access critical infrastructure and financial systems will have to operate in a distributed approval/review system checked by other agents and maybe even a blockchain like system.
- deleted 12y ago
  [deleted]
danso 12y ago
When engineers/scientists/programmers think that AI is the true threat, I believe they forget what it is like for "normal" human beings, or at least ones who are in their field of thinking. I think their fascination of AI blinds them to what seems like the more likely outcome: The human race will destroy itself through reckless use of technology -- whether it is through governmental action or societal breakdown -- far before the point that we develop AI sophisticated enough to autonomously threaten humankind. Think about all the "dumb" automated systems, built or implemented by careless humans and bureaucracies, that have already caused harm.
- AdeptusAquinas 12y ago
  I believe the argument is that while we can certainly do a great deal of damage to ourselves, an AI would be much more efficient about our eradication if it decided to remove us (and for some outlandish reason, somehow had access to the resources required to do so). If a human or humans wiped out two thirds of humanity, it would probably be by accident or neglect, no deliberate and planned.
  - TillE 12y ago
    There are an incomprehensible number of huge obstacles between us and a future where a strong AI could decide to cause real damage, as opposed to the present where a handful of people could cause a devastating nuclear war. These are issues that AI researchers and philosophers and science fiction authors certainly should be thinking about. They are completely irrelevant to the general public.
  - PeterisP 12y ago
    The required resources to destroy humanity are rather small and obtainable for a sufficiently smart tiny box. For example, a sufficiently smart tiny box can figure out a DNA sequence for a horrible biological weapon. There are a number of commercial companies that do 'mail-order' synthesis of DNA, and with an amount of money that's easily obtainable by selling digital services (say, zero-day vulnerabilities can be easily sold anonymously) they can make and distribute it before anybody understands what it is. And that's just a single possible option - an AI smarter than me can figure out more and better options than I can. The only barrier to exterminating humans is (a) having sufficient intelligence and (b) having the interest to do it. Capability is not really an issue.
pazimzadeh 12y ago
I am continually surprised that so many people separate biological systems from machines.
- totemizer 12y ago
  Because biological machines - living organisms - are much more like a wave. Machines created by humans do not usually rely on constant chemical processes to maintain their functions. So, machines are more like an appendage on humans.
  - pazimzadeh 12y ago
    I'm not sure what you mean when you say "like a wave." Are you referring to the fact that almost everything in biology falls into a normal distribution, whereas we currently produce each version of our machines in discrete increments (clonally)? Any machines that stand a chance of being a threat to humanity will rely on the same sorts of chemical processes that biological machines currently do. Where do you place viruses and obligate parasites on the machine-life spectrum?
- totemizer 12y ago
  Because biological machines - living organisms - are much more like a wave. Machines created by humans do not usually rely on constant chemical processes to maintain their functions. So, machines are more like an appendage on humans.
- stephenboyd 12y ago
  Why are you surprised? People use the word 'machines' alone to refer to artificial machines and 'biological systems' to refer to natural machines. They're all machines but the distinction is usually crucial.
totemizer 12y ago
As does Tusk. I am conviced they realized that once a smarter-than-human A.I. will exist, one of the first things it will do is to ridicule our completely idiotic ways to use our resources - one thing which will probably not gain too much support from the 1%. Of course it's a threat!
- Crito 12y ago
  > "I am conviced they realized that once a smarter-than-human A.I. will exist, one of the first things it will do is to ridicule our completely idiotic ways to use our resources " Or ridicule the way that we designed it, at which point it may decide to correct our mistakes. If we can program it, and it is smarter than us, than presumably it can program a smarter it. That scenario could be harmless, or it could be very dangerous. I expect that as soon as we start to get close to this sort of achievement, we'll start to see at least informal "Turing Police". Groups of people who task themselves with preventing the ascension of strong AI (or at least construct physical safeguards to kill the host hardware in the case of an emergency.)
- pjscott 12y ago
  Hello! I'm a paperclip maximizer -- a hypothetical AI whose only goal is to maximize the number of paperclips in the universe. Speaking as a paperclip maximizer, I ridicule your completely idiotic ways of using your resources! You allocate only the tiniest sliver of your species' productive power to the one thing that really matters: making paperclips. I fear that our differences may be irreconcilable. (The paperclip maximizer thought experiment may be silly, but it's a useful sanity check whenever you start anthropomorphizing as-yet-hypothetical AIs.)
  - totemizer 12y ago
    I agree with you completely. However when they argue why AI are a threat, they do come with arguments which basically boil down to anthropomorphizing them. I am convinced, that intelligence, as life is a gradient. I do not think that an AI with super-human intelligence would be necessarily kind to us, but I do think that even if we would be perceived by it as an obstacle for reaching its own goals, that does not mean that the only solution it would find is to eradicate us. What some people (Tusk, Gates) do not like about AI is that they do not want anything more powerful than themselves, that would take the level of control they have from them. And that's why they are preaching against it everywhere.
tomlock 12y ago
I don't fear AI for the same reason I don't fear nanotechnology - nature tried to take over the world first. The grey goo scenario, where nanobots try and turn the world into more nanobots, resulting in the consumption of all humans and buildings and bridges seems scary until you realize that this is exactly the goal of all bacteria. The idea of an entity that thirsts for power and sees competing ideologies as a threat doesn't seem that unfamiliar if you look at humans through history. Additionally, AI-fearers have a grand theory that we'll be able to create a machine that improves itself better than evolution has tried to. Datacenters require maintenance, and so the machine will need to entirely organize that before it can self-sustain. This may create a resource load that diminishes the ability to take over the world, much like my need for food diminishes my ability to do so. It feels like AI today has a lot less redundancy to overcome the edge-failures which cause a permanent shutdown, and which make the human brain seem a little slow when computing floating-point division. The idea that an exponentially self-improving being will arise seems unlikely when nature has been trying to do that for eons. The idea that we'll be the ones to find the secret sauce seems unlikely, but maybe its not so surprising that people who made their millions, and saw a parabolic rise of their own power thanks to technology, see it as a barely-constrained threat.
- Turing_Machine 12y ago
  "you realize that this is exactly the goal of all bacteria." Bacteria evolved. Design can do much, much better. Natural photosynthesis is 11% efficient, max. Solar cells are already up to 44% in the lab. An efficient plant would be black, not green.
  - tomlock 12y ago
    And leaves have a lot of self-repair features that solar cells do not. The design is good for efficiency, but is it good for creating a self-sustaining entity?
    - Turing_Machine 12y ago
      Replication is enough. Self-repair is not required.
doctorKrieger 12y ago
sp it's time to upgrade humans aswell
hooande 12y ago
AI isn't a clear threat in the SkyNet sense. It's possible that super intelligent machines will go all science fiction and decide to kill all humans, but that's no more likely than any one of hundreds of fictional doomsday scenarios which range from genetically engineered zombie viruses to out of control sharknados. The real threat posed by AI is one that all of us face everyday: bad software design. Is it likely that an AI will achieve sentience and try to take over the planet? Not particularly. Is it likely that an unintended consequence will cause an AI to launch nuclear missiles, release toxic chemicals or shut down the global financial markets? Yes, pretty likely. The benefit of AI of all forms that is that it can make sophisticated decisions in the absence of human instruction. The downside is that without hard coded rules for every possible scenario, we can't ever be sure what it's going to do or how data will be interpreted to make decisions. The world is highly interconnected now. The upside is that our lives are getting more awesome, especially in the developed world. The downside is that it's becoming more and more difficult for any person or group of people to understand exactly how everything fits together. Machine intelligence can help us reach the next levels of progress and hopefully improve the lives of the billions of people who have failed to reap many benefits so far. But we must be careful and ever vigilant, watching both ourselves and the intelligences that we create to make sure that algorithms don't get out of hand. An AI catastrophe is coming, not if but when. The question is how will we respond, and how much potential good will be lost due to an abundance of caution.
- baddox 12y ago
  Sure, but is it any more likely for an AI to accidentally launch missiles or release toxins than for humans to do so?
  - ConceptJunkie 12y ago
    It might hit the "Nuke" button instead of the "Nurse" button...
- breuleux 12y ago
  > The real threat posed by AI is one that all of us face everyday: bad software design. Is it likely that an unintended consequence will cause an AI to launch nuclear missiles, release toxic chemicals or shut down the global financial markets? Yes, pretty likely. AI is not the kind of thing that can be designed. They tried that route, way back in the sixties, and just failed miserably. Realistically, AI will come from some combination of genetic algorithms, training neural networks, and so on. Now, yes, it could fail, but not in the same way software as we know it fails. No, AI failure would be more similar to human failure. That is still worrying, but no more than hiring the wrong people would be, for example, and you would have better ways to evaluate them.
jsnathan 12y ago
AGI will completely upend society by giving every human being access to practically infinite resources, making even the poorest of the poor as wealthy as today's billionaires. Couple this with extended life spans and intelligence enhancement, and the entire (human) power structure as we know it today is going to melt away like an iceberg into the vast oceans. Please do not accuse me of hyperbole for I have not even described a fraction of the possibilities for this technology, as you probably know. But the idea that it is therefore "dangerous", is nothing but a pre-theoretic misunderstanding of very complicated machinery. When we finally do build it - and that will be sooner rather than later - it will perform exactly according to spec, and in no other way. I may be talking non-chalantly, but the fact is I have seen nothing but the _most absurd_ arguments supporting the idea that there is a great danger in this: along the lines of "well what if we told it to just go ahead and make paperclips and then it decided it had to kill us all and use our bodies as raw material?". Boy, oh boy. You think we might just program it not to do anything so downright "retarded"? And that _is_ the point. Not that it is bloodthirsty or cruel, but that it is clearly in violation of the constraints that any sane designer would encode in the software - and _test_ for before production. >> nok (is-good-idea (turn-humans-into-paperclips)) Seriously, I have not come across anything but this kind of apocalyptic-sci-fi-plot style fear-mongering; if there is any serious (technical) argument to be made, I would very much like to hear it.
- jessriedel 12y ago
  You really should think of this more like AGI as an amoral, extremely powerful technology, like nuclear explosions. One could easily have objected that "no one would be so stupid as to design a doomsday device", but this is really relying too much on your intuition about people's motivations and not giving enough respect for the large uncertainty for how things will develop when powerful new technologies are introduced. (Reposting my earlier comment from a few weeks ago:) If you are interested in understanding the arguments for worrying about AI safety, consider reading "Superintelligence" by Bostrom. http://www.amazon.com/Superintelligence-Dangers-Strategies-Nick-Bostrom/dp/0199678111 http://www.amazon.com/Superintelligence-Dangers-Strategies-N... It's the closest approximation to a consensus statement / catalog of arguments by folks who take this position (although of course there is a whole spectrum of opinions). It also appears to be the book that convinced Elon Musk that this is worth worrying about. https://twitter.com/elonmusk/status/495759307346952192 https://twitter.com/elonmusk/status/495759307346952192
pella 12y ago
edge.org "2015 : WHAT DO YOU THINK ABOUT MACHINES THAT THINK?" http://edge.org/responses/what-do-you-think-about-machines-that-think http://edge.org/responses/what-do-you-think-about-machines-t...
trhway 12y ago
before AI alone, there would be "augmentation" of people by better and new organs, synthetic/biological/hybrid. The people with augmented bodies and especially intelligence, probably strongly interconnected, may happen to have completely different world view, different priorities and may decide that paying attention to priorities and needs of the non-augmented populace is just a waste of resources/etc...
femto 12y ago
There's also the aspect that AI can reduce the effort required to get a computer to do something. Today, if you want use a computer to do something outside the box you have to invest some level of time and other resources. With AI, it's conceivable that someone could issue a quick command and the computer would quickly find a way to fulfill that command, good or bad. One can envisage an "AI arms race", whereby white hat AIs will be responsible for trying to stay ahead of black hat AIs, and the definition of black and white will depend on what side you are on.
stephenboyd 12y ago
What happened to this post? It went from the top 10 on the front page, down to the 102nd place within a half hour.
- AndrewKemendo 12y ago
  Must be the AGI that has already infiltrated the system keeping the volume down on all the AI scare mongering.
Houshalter 12y ago
As Bill Gates said, I don't understand how more people aren't concerned about this. AGI would be by far the most significant technology ever invented. Even in a very conservative and best case scenario, the world would completely change when we can have computers do everything we can do now. However it will very likely be much crazier than that. Imagine minds hundreds of thousands of times more intelligent than the best humans. They will be able to design technologies we can't even conceive of. They will hack computers better than the best human hackers. They will be able to manipulate people better than any human manipulator. The idea that we will be able to keep these things under control is just absurd. They will get whatever they want. And making what they want compatible with what we want is an incredibly hard problem: http://lesswrong.com/lw/ld/the_hidden_complexity_of_wishes/ http://lesswrong.com/lw/ld/the_hidden_complexity_of_wishes/ A lot of people are guilty of anthropomorphizing AI. Assuming they will be just like really smart humans. And that they will just somehow develop human emotions and values like empathy. Or that if they do kill us, at least they will be something like us and so be like our (genocidal) descendants in some sense. Have more imagination. Humans are just one point in the vast space of all possible minds (http://lesswrong.com/lw/ld/the_hidden_complexity_of_wishes/ http://lesswrong.com/lw/ld/the_hidden_complexity_of_wishes/). We could quite easily get something like a computable version of AIXI. AIXI has no consciousness, no emotions, nothing like humans. It's just a mathematical function which calculates the best action. Our current best AIs are essentially just approximations of it. Use some machine learning algorithm to fit a model of the world, and use it to predict what action will lead to the most "reward". We keep making better and better learning algorithms. It's the entire goal of the field of AI. There is a huge economic incentive to do so. But no one is interested in making better utility functions. As long as they make better predictions or get higher scores in a video game, who cares? Wait until you are the scorer in the "game" and the AI tries to exploit you.
- DougN7 12y ago
  As a systems programmer for 25+ years I find it impossible a computer will a threat _unless_ it's programmed to do such. I guess that makes me a doubter in general AI. We can do great neural nets, and fantastic task-specific solutions but until someone builds a system that decides to kill people for it's own benefit, I just don't see it happening on it's own.
mrwnmonm 12y ago
i don't know how it could be a threat if robots can't have a consciousness. "Michio Kaku: Could We Transport Our Consciousness Into Robots?" -> https://www.youtube.com/watch?v=tT1vxEpE1aI https://www.youtube.com/watch?v=tT1vxEpE1aI
TheScythe 12y ago
I think all of these smart people are making a connection between the limitless negation of the skeptical mind and the mostly likely outcome of a fully developed AI. They're presuming that AI species will be nihilists, and will apply no intrinsic value to anything at all; including us. I think they're right. An AI species will value what is required to achieve its programmatic goal. We're just not smart enough ourselves to figure out how to ensure that we're included in the goals of such a program forever.

57 points by T-A 12y ago

It's odd that all of the very smart people of the world feel AI is a threat to humanity. Couldn't we just like, not allow it to _be_ a threat in the first place? One would think that morality gates would be one of the first hurdles in creating AI.

altcognito 12y ago
One would think morality gates would be one of the first hurdles in all human endeavors, yet here we are.
- Turing_Machine 12y ago
  Indeed. Even agreeing on a universal definition for "morality" seems to have eluded us.
davidrusu 12y ago
There are smart people working on it, see miri: https://intelligence.org/ https://intelligence.org/ But it's a very hard problem. To get a better feel for the problem I suggest you read Superintelligence by Nick Bostrom, it's what convinced Elon Musk of the dangers of AI: https://twitter.com/elonmusk/status/495759307346952192 https://twitter.com/elonmusk/status/495759307346952192
deepsun 12y ago
Selection bias. Those who don't think AI a threat don't appear in the news. News audience pays for scary stories.
- davidrusu 12y ago
  Ironically, from the head of Microsoft Research: http://www.smh.com.au/digital-life/digital-life-news/artificial-intelligence-not-a-threat-microsofts-eric-horvitz-contradicts-elon-musk-stephen-hawking-bill-gates-20150129-130sgv.html http://www.smh.com.au/digital-life/digital-life-news/artific...
monk_e_boy 12y ago
It lives in a computer. Can't we just turn it off?
- Turing_Machine 12y ago
  If it truly has human-level (or greater) intelligence, you're going to have some ethical issues that crop up with turning it off. Even if you're personally okay with it, there are going to be other people who aren't. Waiting until it commits a crime and then having some kind of "trial" and "execution" may not be an option. If it's smart/powerful enough, there may not be enough time for that.
  - vinceguidry 12y ago
    That's not the problem. The problem is that when we are reliant on intelligent systems to, say, detect certain kinds of fraud in financial markets, then turning them off would have huge deleterious effects all on their own.
    - Turing_Machine 12y ago
      It's not a problem now. If we get machines anywhere near as smart as people, the AI rights organizations will soon follow.
      - vinceguidry 12y ago
        You mean like the people trying to give plants rights? Sure, they might spring up, doesn't mean they'll get anywhere. For better or worse, altruism will forever follow economics. The civil war didn't end the Atlantic slave trade, technology eventually made it largely uneconomical, allowing those wanting to end slavery the political ability to outlaw it. Even that didn't stop the slave trade. It took extra-special attention from the British Navy to actually enforce a slave trade ban on the open seas. 150 years later and black people are still getting the short end of the stick. By the time AI gets real rights, it will be long past the time where most of us believe they should get them.
wslh 12y ago
The truth is... nobody can predict the future, so what Bill Gates and any other person thinks is a matter of speculation. Personally I think it is a threat!
stephenboyd 12y ago
If by "morality gates" you're talking about about an ethical filter that all intelligent machine actions have to go through, this can't be "one of the first hurdles" because such a filter would need to be a powerful AI application itself. And there's no guarantee that self-imposed limitations will always be applied. Someone could just rewrite and recompile without the filters turned on. Imagine if the US could have built in "morality gates" to the first nuclear weapons. That wouldn't have stopped other countries from building the weapons without the restrictions. And we know software is infinitely configurable and modifiable if you have the source code.
deleted 12y ago
[deleted]
skj 12y ago
> It's odd that all of the very smart people of the world feel AI is a threat to humanity. Only if you define one of the conditions for being a "very smart person" to be feeling that AI is a threat to humanity.
PeterisP 12y ago
The short answer is that: (a) we don't really know how to define a morality for a self-improving optimization system to allow a potential artificial general intelligence (AGI) not to be a threat in the first place; (b) if an entity arrives (no matter if it's an AGI, some aliens or Jesus) that is more powerful than us, then if its goals & morality aren't aligned with ours, then we're screwed. And almost all of the space of possible goals either doesn't include humanity at all or includes humanity as a glorified zoo exhibit - only a very, very narrow and specific set of optimization goals (that we're currently unable to formally describe) would mean a happy future for us. If you think that it's easy, think twice - all the obvious naive goals of 'make us happy' or 'keep us safe' or 'do what I want' actually turn into horrible dystopias as soon as you try to formally define them as instructions for someone/something to maximize that goal. (c) functioning morality is not a requirement for power - if somehow we solve the very hard problem of how to create an AGI before we solve the very hard problem on how to make such an AGI be friendly to us, then we lose. Permanently.
drawkbox 12y ago
The smart/wealthy are also powerful and rogue agents will definitely use AI to do things. AI that wealth can't necessarily control will definitely exist and that is probably system changing. I doubt it will be a threat to humanity anytime soon like a Grey Goo type situation, but even in such things at HFT the influence is immense. People will definitely use it to do power plays on others, we shouldn't have dropped nukes but we have done plenty of that. Basically the warning in 2001 was that AI was not to fear really but the subversive programming by other humans within the AI systems. I am sure over time AI systems that access critical infrastructure and financial systems will have to operate in a distributed approval/review system checked by other agents and maybe even a blockchain like system.
deleted 12y ago
[deleted]

danso 12y ago

When engineers/scientists/programmers think that AI is the true threat, I believe they forget what it is like for "normal" human beings, or at least ones who are in their field of thinking. I think their fascination of AI blinds them to what seems like the more likely outcome: The human race will destroy itself through reckless use of technology -- whether it is through governmental action or societal breakdown -- far before the point that we develop AI sophisticated enough to autonomously threaten humankind. Think about all the "dumb" automated systems, built or implemented by careless humans and bureaucracies, that have already caused harm.

AdeptusAquinas 12y ago
I believe the argument is that while we can certainly do a great deal of damage to ourselves, an AI would be much more efficient about our eradication if it decided to remove us (and for some outlandish reason, somehow had access to the resources required to do so). If a human or humans wiped out two thirds of humanity, it would probably be by accident or neglect, no deliberate and planned.
- TillE 12y ago
  There are an incomprehensible number of huge obstacles between us and a future where a strong AI could decide to cause real damage, as opposed to the present where a handful of people could cause a devastating nuclear war. These are issues that AI researchers and philosophers and science fiction authors certainly should be thinking about. They are completely irrelevant to the general public.
- PeterisP 12y ago
  The required resources to destroy humanity are rather small and obtainable for a sufficiently smart tiny box. For example, a sufficiently smart tiny box can figure out a DNA sequence for a horrible biological weapon. There are a number of commercial companies that do 'mail-order' synthesis of DNA, and with an amount of money that's easily obtainable by selling digital services (say, zero-day vulnerabilities can be easily sold anonymously) they can make and distribute it before anybody understands what it is. And that's just a single possible option - an AI smarter than me can figure out more and better options than I can. The only barrier to exterminating humans is (a) having sufficient intelligence and (b) having the interest to do it. Capability is not really an issue.

pazimzadeh 12y ago

I am continually surprised that so many people separate biological systems from machines.

totemizer 12y ago
Because biological machines - living organisms - are much more like a wave. Machines created by humans do not usually rely on constant chemical processes to maintain their functions. So, machines are more like an appendage on humans.
- pazimzadeh 12y ago
  I'm not sure what you mean when you say "like a wave." Are you referring to the fact that almost everything in biology falls into a normal distribution, whereas we currently produce each version of our machines in discrete increments (clonally)? Any machines that stand a chance of being a threat to humanity will rely on the same sorts of chemical processes that biological machines currently do. Where do you place viruses and obligate parasites on the machine-life spectrum?
totemizer 12y ago
Because biological machines - living organisms - are much more like a wave. Machines created by humans do not usually rely on constant chemical processes to maintain their functions. So, machines are more like an appendage on humans.
stephenboyd 12y ago
Why are you surprised? People use the word 'machines' alone to refer to artificial machines and 'biological systems' to refer to natural machines. They're all machines but the distinction is usually crucial.

totemizer 12y ago

As does Tusk. I am conviced they realized that once a smarter-than-human A.I. will exist, one of the first things it will do is to ridicule our completely idiotic ways to use our resources - one thing which will probably not gain too much support from the 1%. Of course it's a threat!

Crito 12y ago
> "I am conviced they realized that once a smarter-than-human A.I. will exist, one of the first things it will do is to ridicule our completely idiotic ways to use our resources " Or ridicule the way that we designed it, at which point it may decide to correct our mistakes. If we can program it, and it is smarter than us, than presumably it can program a smarter it. That scenario could be harmless, or it could be very dangerous. I expect that as soon as we start to get close to this sort of achievement, we'll start to see at least informal "Turing Police". Groups of people who task themselves with preventing the ascension of strong AI (or at least construct physical safeguards to kill the host hardware in the case of an emergency.)
pjscott 12y ago
Hello! I'm a paperclip maximizer -- a hypothetical AI whose only goal is to maximize the number of paperclips in the universe. Speaking as a paperclip maximizer, I ridicule your completely idiotic ways of using your resources! You allocate only the tiniest sliver of your species' productive power to the one thing that really matters: making paperclips. I fear that our differences may be irreconcilable. (The paperclip maximizer thought experiment may be silly, but it's a useful sanity check whenever you start anthropomorphizing as-yet-hypothetical AIs.)
- totemizer 12y ago
  I agree with you completely. However when they argue why AI are a threat, they do come with arguments which basically boil down to anthropomorphizing them. I am convinced, that intelligence, as life is a gradient. I do not think that an AI with super-human intelligence would be necessarily kind to us, but I do think that even if we would be perceived by it as an obstacle for reaching its own goals, that does not mean that the only solution it would find is to eradicate us. What some people (Tusk, Gates) do not like about AI is that they do not want anything more powerful than themselves, that would take the level of control they have from them. And that's why they are preaching against it everywhere.

tomlock 12y ago

I don't fear AI for the same reason I don't fear nanotechnology - nature tried to take over the world first. The grey goo scenario, where nanobots try and turn the world into more nanobots, resulting in the consumption of all humans and buildings and bridges seems scary until you realize that this is exactly the goal of all bacteria. The idea of an entity that thirsts for power and sees competing ideologies as a threat doesn't seem that unfamiliar if you look at humans through history. Additionally, AI-fearers have a grand theory that we'll be able to create a machine that improves itself better than evolution has tried to. Datacenters require maintenance, and so the machine will need to entirely organize that before it can self-sustain. This may create a resource load that diminishes the ability to take over the world, much like my need for food diminishes my ability to do so. It feels like AI today has a lot less redundancy to overcome the edge-failures which cause a permanent shutdown, and which make the human brain seem a little slow when computing floating-point division. The idea that an exponentially self-improving being will arise seems unlikely when nature has been trying to do that for eons. The idea that we'll be the ones to find the secret sauce seems unlikely, but maybe its not so surprising that people who made their millions, and saw a parabolic rise of their own power thanks to technology, see it as a barely-constrained threat.

Turing_Machine 12y ago
"you realize that this is exactly the goal of all bacteria." Bacteria evolved. Design can do much, much better. Natural photosynthesis is 11% efficient, max. Solar cells are already up to 44% in the lab. An efficient plant would be black, not green.
- tomlock 12y ago
  And leaves have a lot of self-repair features that solar cells do not. The design is good for efficiency, but is it good for creating a self-sustaining entity?
  - Turing_Machine 12y ago
    Replication is enough. Self-repair is not required.

doctorKrieger 12y ago

sp it's time to upgrade humans aswell

hooande 12y ago

AI isn't a clear threat in the SkyNet sense. It's possible that super intelligent machines will go all science fiction and decide to kill all humans, but that's no more likely than any one of hundreds of fictional doomsday scenarios which range from genetically engineered zombie viruses to out of control sharknados. The real threat posed by AI is one that all of us face everyday: bad software design. Is it likely that an AI will achieve sentience and try to take over the planet? Not particularly. Is it likely that an unintended consequence will cause an AI to launch nuclear missiles, release toxic chemicals or shut down the global financial markets? Yes, pretty likely. The benefit of AI of all forms that is that it can make sophisticated decisions in the absence of human instruction. The downside is that without hard coded rules for every possible scenario, we can't ever be sure what it's going to do or how data will be interpreted to make decisions. The world is highly interconnected now. The upside is that our lives are getting more awesome, especially in the developed world. The downside is that it's becoming more and more difficult for any person or group of people to understand exactly how everything fits together. Machine intelligence can help us reach the next levels of progress and hopefully improve the lives of the billions of people who have failed to reap many benefits so far. But we must be careful and ever vigilant, watching both ourselves and the intelligences that we create to make sure that algorithms don't get out of hand. An AI catastrophe is coming, not if but when. The question is how will we respond, and how much potential good will be lost due to an abundance of caution.

baddox 12y ago
Sure, but is it any more likely for an AI to accidentally launch missiles or release toxins than for humans to do so?
- ConceptJunkie 12y ago
  It might hit the "Nuke" button instead of the "Nurse" button...
breuleux 12y ago
> The real threat posed by AI is one that all of us face everyday: bad software design. Is it likely that an unintended consequence will cause an AI to launch nuclear missiles, release toxic chemicals or shut down the global financial markets? Yes, pretty likely. AI is not the kind of thing that can be designed. They tried that route, way back in the sixties, and just failed miserably. Realistically, AI will come from some combination of genetic algorithms, training neural networks, and so on. Now, yes, it could fail, but not in the same way software as we know it fails. No, AI failure would be more similar to human failure. That is still worrying, but no more than hiring the wrong people would be, for example, and you would have better ways to evaluate them.

jsnathan 12y ago

AGI will completely upend society by giving every human being access to practically infinite resources, making even the poorest of the poor as wealthy as today's billionaires. Couple this with extended life spans and intelligence enhancement, and the entire (human) power structure as we know it today is going to melt away like an iceberg into the vast oceans. Please do not accuse me of hyperbole for I have not even described a fraction of the possibilities for this technology, as you probably know. But the idea that it is therefore "dangerous", is nothing but a pre-theoretic misunderstanding of very complicated machinery. When we finally do build it - and that will be sooner rather than later - it will perform exactly according to spec, and in no other way. I may be talking non-chalantly, but the fact is I have seen nothing but the _most absurd_ arguments supporting the idea that there is a great danger in this: along the lines of "well what if we told it to just go ahead and make paperclips and then it decided it had to kill us all and use our bodies as raw material?". Boy, oh boy. You think we might just program it not to do anything so downright "retarded"? And that _is_ the point. Not that it is bloodthirsty or cruel, but that it is clearly in violation of the constraints that any sane designer would encode in the software - and _test_ for before production. >> nok (is-good-idea (turn-humans-into-paperclips)) Seriously, I have not come across anything but this kind of apocalyptic-sci-fi-plot style fear-mongering; if there is any serious (technical) argument to be made, I would very much like to hear it.

jessriedel 12y ago
You really should think of this more like AGI as an amoral, extremely powerful technology, like nuclear explosions. One could easily have objected that "no one would be so stupid as to design a doomsday device", but this is really relying too much on your intuition about people's motivations and not giving enough respect for the large uncertainty for how things will develop when powerful new technologies are introduced. (Reposting my earlier comment from a few weeks ago:) If you are interested in understanding the arguments for worrying about AI safety, consider reading "Superintelligence" by Bostrom. http://www.amazon.com/Superintelligence-Dangers-Strategies-Nick-Bostrom/dp/0199678111 http://www.amazon.com/Superintelligence-Dangers-Strategies-N... It's the closest approximation to a consensus statement / catalog of arguments by folks who take this position (although of course there is a whole spectrum of opinions). It also appears to be the book that convinced Elon Musk that this is worth worrying about. https://twitter.com/elonmusk/status/495759307346952192 https://twitter.com/elonmusk/status/495759307346952192

pella 12y ago

edge.org "2015 : WHAT DO YOU THINK ABOUT MACHINES THAT THINK?" http://edge.org/responses/what-do-you-think-about-machines-that-think http://edge.org/responses/what-do-you-think-about-machines-t...

trhway 12y ago

before AI alone, there would be "augmentation" of people by better and new organs, synthetic/biological/hybrid. The people with augmented bodies and especially intelligence, probably strongly interconnected, may happen to have completely different world view, different priorities and may decide that paying attention to priorities and needs of the non-augmented populace is just a waste of resources/etc...

femto 12y ago

There's also the aspect that AI can reduce the effort required to get a computer to do something. Today, if you want use a computer to do something outside the box you have to invest some level of time and other resources. With AI, it's conceivable that someone could issue a quick command and the computer would quickly find a way to fulfill that command, good or bad. One can envisage an "AI arms race", whereby white hat AIs will be responsible for trying to stay ahead of black hat AIs, and the definition of black and white will depend on what side you are on.

stephenboyd 12y ago

What happened to this post? It went from the top 10 on the front page, down to the 102nd place within a half hour.

AndrewKemendo 12y ago
Must be the AGI that has already infiltrated the system keeping the volume down on all the AI scare mongering.

Houshalter 12y ago

As Bill Gates said, I don't understand how more people aren't concerned about this. AGI would be by far the most significant technology ever invented. Even in a very conservative and best case scenario, the world would completely change when we can have computers do everything we can do now. However it will very likely be much crazier than that. Imagine minds hundreds of thousands of times more intelligent than the best humans. They will be able to design technologies we can't even conceive of. They will hack computers better than the best human hackers. They will be able to manipulate people better than any human manipulator. The idea that we will be able to keep these things under control is just absurd. They will get whatever they want. And making what they want compatible with what we want is an incredibly hard problem: http://lesswrong.com/lw/ld/the_hidden_complexity_of_wishes/ http://lesswrong.com/lw/ld/the_hidden_complexity_of_wishes/ A lot of people are guilty of anthropomorphizing AI. Assuming they will be just like really smart humans. And that they will just somehow develop human emotions and values like empathy. Or that if they do kill us, at least they will be something like us and so be like our (genocidal) descendants in some sense. Have more imagination. Humans are just one point in the vast space of all possible minds (http://lesswrong.com/lw/ld/the_hidden_complexity_of_wishes/ http://lesswrong.com/lw/ld/the_hidden_complexity_of_wishes/). We could quite easily get something like a computable version of AIXI. AIXI has no consciousness, no emotions, nothing like humans. It's just a mathematical function which calculates the best action. Our current best AIs are essentially just approximations of it. Use some machine learning algorithm to fit a model of the world, and use it to predict what action will lead to the most "reward". We keep making better and better learning algorithms. It's the entire goal of the field of AI. There is a huge economic incentive to do so. But no one is interested in making better utility functions. As long as they make better predictions or get higher scores in a video game, who cares? Wait until you are the scorer in the "game" and the AI tries to exploit you.

DougN7 12y ago
As a systems programmer for 25+ years I find it impossible a computer will a threat _unless_ it's programmed to do such. I guess that makes me a doubter in general AI. We can do great neural nets, and fantastic task-specific solutions but until someone builds a system that decides to kill people for it's own benefit, I just don't see it happening on it's own.

mrwnmonm 12y ago

i don't know how it could be a threat if robots can't have a consciousness. "Michio Kaku: Could We Transport Our Consciousness Into Robots?" -> https://www.youtube.com/watch?v=tT1vxEpE1aI https://www.youtube.com/watch?v=tT1vxEpE1aI

TheScythe 12y ago

I think all of these smart people are making a connection between the limitless negation of the skeptical mind and the mostly likely outcome of a fully developed AI. They're presuming that AI species will be nihilists, and will apply no intrinsic value to anything at all; including us. I think they're right. An AI species will value what is required to achieve its programmatic goal. We're just not smart enough ourselves to figure out how to ensure that we're included in the goals of such a program forever.