9 ms·

Stephen Hawking: AI, Potentially the Worst Thing to Happen to Humanity

66 points by jaequery 12y ago

DigitalSea 12y ago
I share Stephen Hawking's concerns. I think AI is already at a point where maybe 10 or even 15 years down the road, artificial intelligence will be everywhere. Autonomous public transport, taxis, driver-less vehicles, autonomous crime prevention systems. The military is said to always be at least 10 years ahead of public tech research and development, look what they've already got now, imagine another 10 years. We've become a society that is too big and intelligent to fail, but that ignorance of being too smart and big to fail could be that our downfall is.
- sidcool 12y ago
  I share the concern partially. It's similar to other technologies which have downsides. Nuclear power for example. AI will have the power to help humanity prosper or cause its downfall. The balance will be for us to decide, again just like Nuclear power; for with great power comes great responsibility.
owens99 12y ago
Anyone know of a larger discussion related to this issue?
- timClicks 12y ago
  The Less Wrong community, various subreddits (/r/singularity, etc). Good terms to search for are "x-risk", "Jaan Tallinn" and "Nick Bostrom".
deleted 12y ago
[deleted]
- eurleif 12y ago
  Somehow, in the context of Stephen Hawking, this joke seems a bit insensitive.
  - deleted 12y ago
    [deleted]
deleted 12y ago
[deleted]
deleted 12y ago
[deleted]
bad_alloc 12y ago
I recommend reading "Shadows of the Mind" by Roger Penrose. In that book he argues that strong artificial intelligence isn't achievable on classical computers, since the human brain seems to have the ability to do quantum computing in certain molecular structures. These processes probably can't be simulated efectively. So AI in today's computers probably won't work at all (although quantum computers or artificial neural networks might).
- deleted 12y ago
  [deleted]
- dlss 12y ago
  This has been discredited (along with Penrose's incompleteness theorem based "AI is impossible" conjecture before it) http://en.wikipedia.org/wiki/Quantum_mind#Roger_Penrose_and_Stuart_Hameroff http://en.wikipedia.org/wiki/Quantum_mind#Roger_Penrose_and_...
  - ronaldx 12y ago
    Although specific claims might have been discredited, quantum processes are relevant to biology, e.g. http://www.nature.com/nature/journal/v446/n7137/abs/nature05678.html http://www.nature.com/nature/journal/v446/n7137/abs/nature05...
    - dlss 12y ago
      This is true, and it's also true that such quantum processes exist in the brain. Penrose's claim was that those quantum processes are a significant part of the computation that is consciousness. It was discredited by getting better upper bounds on how much quantum computation the brain is capable of (ie very little).
harshreality 12y ago
This is what MIRI (Machine Intelligence Research Institute) is investigating. http://intelligence.org/research/ http://intelligence.org/research/ The scariest part of the problem to me is this: http://yudkowsky.net/singularity/aibox http://yudkowsky.net/singularity/aibox If an AI can't be contained, then there has to be some failsafe that prevents any self-modifications which could turn it evil OR remove the failsafe. Or you have to believe that you can neutralize an evil AI before it destroys humanity.
- deutronium 12y ago
  Regarding the AI box experiment, I'm somewhat dubious about it, considering there doesn't seem to be any transcripts from when the AI 'escapes'. Which makes me think they use blackmail or something similar in the transcript.
  - danbruc 12y ago
    I couldn't imagine how the AI could escape an hour ago, but after thinking about it for a while I am more and more convinced that it is actually hard to contain the AI. Containing the AI is like imprisoning a human for no reason and this will usually conflict with the gatekeepers moral values. Just imagine talking to human suffering in a tiny prison cell without having done anything wrong instead of the trapped AI. Can you see how it would be the right thing to open the cell's door? You are cruel not opening the door.
  - jimrandomh 12y ago
    The AI experiment reproduced by Tuxedage: http://lesswrong.com/lw/gej/i_attempted_the_ai_box_experiment_and_lost/ http://lesswrong.com/lw/gej/i_attempted_the_ai_box_experimen... . 3 wins and 3 losses.
- Udo 12y ago
  The AIBox link contains a lot of buildup but ultimately delivers nothing. All we get is two people saying after a while "I let it out", questions as to the details of what happened are just not answered. I'm not denying it could be a fun game to play, but the article makes it way too painful and timeconsuming to find out what you're reading is in fact just the description of a game, and not an example for a strategy that could be used to let the AI out. That being said, yes, what's the point of having an AGI if you're not letting it "out"? It will probably escape anyway, so containment is not a solution. By the way, there is a lot of precedent in human history about the effectiveness of containing "dangerous" ideas by imprisoning the people who developed them. It generally doesn't work out. If you're developing AGI, you've pretty much already resigned to the idea that the world will change. The destruction of the human race is only one extreme outcome. Especially if the takeoff isn't very hard, there are a lot of scenarios where a net-beneficial AI could be developed. And all the engineering experience we have does indeed point to a very soft takeoff. Ultimately, I think the goal is NOT to generate a slave race with super powers. Yet that's what a lot of people are talking about. It's not only morally wrong but also self-evidently stupid. A lot of influential futurists are fans of that idea though, including Kurzweil. Useful AGI scenarios range from mutually beneficial coexistence to outright merging with the AGIs, and I think that's a goal well worth pursuing. Another strategy would be to keep augmenting ourselves and to continue on a transhumanization path until we are the AGI. You could indeed make an argument that's what's already happening.
daemonk 12y ago
What would the AI's motivation be? Will it understand what existence means and have the "desire" to survive? Is it possible that a self aware AI has already occurred in some form many times in the past, but since it doesn't have the anthropomorphic trait of wanting to exist, it never actively persisted? So what we are really afraid of is developing an AI with human motivations. But do the intelligence we want to artificially develop require human motivations? Aren't they two mutually exclusive traits? Maybe part of the fear is how we train the AI. If we are to imbue it with the total knowledge of our history, then maybe it will become more "human" and have those motivations. Is it possible to train it with hard data that aren't anthropomorphized in any way?
- marvin 12y ago
  An AI's motivation would be whatever it is programmed to do. Which is why it is so important to get this right the first time. Human motivations are incredibly complex, human moral systems have not yet been formalized and any attempt at doing this breaks down at the edge cases. This is not by any means a solved question, but there have been quite a few publications on this already. There are a hundred subtle errors of reasoning that ruin naïve and anthropocentric reasoning around morality and AI safety. Have a look at the lesswrong blog, for instance. Lots of interesting reading.
  - sampo 12y ago
    > "An AI's motivation would be whatever it is programmed to do." Well we don't know whether it is possible to build systems as complex as human brain, and at the same time give them some specific goals.
  - jeremiah256 12y ago
    If human motivations can be quantified and programmed into an AI system, my guess is 'loyalty' or 'patriotism' will be one of the first. "It is lamentable, that to be a good patriot one must become the enemy of the rest of mankind." Voltaire
- andreyf 12y ago
  I find it a rather convincing idea that human motivations are embodied in a sense that would take a lot of effort and no discernible purpose to replicate in an artificial system. I think more interesting is what Hawkins dismisses as "short term implications": who will control the intelligence enhancing software that we will see within our lifetimes and for what purpose? One example: what's up with these marketing+tech giants and what is their end-game? How accurately will we be able model desires/fears/motivations and how well can those with access to those models guide a person's beliefs and desires?
javindo 12y ago
I don't really want to come across as hostile to the views of such a well respect physicist, but this article represents the view which is often held by those who are very intelligent yet lack the actual study of AI. Anybody who has actually, or is currently, studying AI will know that there are fundamental differences in programming "AI" and some self aware magic computation which is almost entirely unfathomable. With current knowledge, at least, the only way to have any sort of learning is to design and implement algorithms to do so. There can of course be incredibly abstract knowledge base designs and the computation can assimilate incredibly complex knowledge from those, however it is almost a logical fallacy to suggest that we would ever lose control of something autonomous due to some sort of "rogue agent". Of course, there are very real risks with something like AI, but it's much less sinister than what the article suggests. For example, faulty code or errors in the knowledge base could lead it to make a bad decision, but humans can also do that. I don't think we should be dismissing the risks necessarily, but I do believe that it is completely far fetched to say we would be making the worst mistake in history if we are to dismiss a film as science fiction!
- marvin 12y ago
  On my side, I am surprised that so many AI researchers are unable to take the long-term view in this discussion. It's presumably because AI in the laboratory is still relatively primitive. No one is talking about magic. The human brain is not magic, neither is that of chimpanzees, rats, dolphins or gorillas. Intelligence is a purely physical phenomenon, which means it can be emulated by computers. Natural brains are also a product of evolution, which means (1) the development happens very slowly, (2) development is directed only towards evolutionary success and (3) there is no flexibility in how the thinking organs are constructed. Computer intelligence does not in principle have these limitations. It would be terribly anthropocentric to believe that humans are the most sophisticated intelligent entity that can exist in the physical world - after all, we are as far as we know the first such entity to emerge, so from our perspective the evolution of intelligence has now stopped. That's the feasability argument. The risk argument is that the consequences of an independent, runaway intelligent entity significantly more capable than humans would have such devastating consequences for humanity's future that even a small risk merits a significant effort to map out the territory. Respected scientists have said "it is impossible" to hundreds of things that proved to be quite simple, so this is not an argument. Even if you don't buy the previous part of my reasoning, there is still risk here. In principle, there are any number of things that could preclude advanced AI in the near future even if the above reasoning is correct (too difficult, requires too much computing power, uses different computational techniques), but seeing as we don't know the unknowns here, taking the cautious route is the correct thing to do. This has been a scientific principle for decades, no reason to drop now. Do AI researchers have any arguments opposing this that don't amount to "the AI we have created up until now is not very good"?
nanomage 12y ago
so, Mr. Hawking hasseen 'The Animetrix,' I gather?
tim333 12y ago
Assuming strong AI arises, which I think likely, and given human nature I imagine people will build all sorts of AI's, 'evil' and 'good', though those terms are a bit subjective. Probably the likes of Kim Jong Un would love to have an army of self reproducing killer robots and Google will try to build nice ones and the US defence department will continue with drone like things. I agree with Hawking it could be a problem and hope the nice ones win out as they probably will. Fingers crossed.
tmikaeld 12y ago
Simplistically: If Win = Continue If Loose = Change Basic evolutionary principle. Then giving the tools of doing, would be required to consume the formula. I argue that an AI with limitations would not work.
grondilu 12y ago
Even if AI is harmless, there is still the risk of brain emulation. To me this is a greater threat, because if we don't know how a fully artificial mind would behave, we can get an idea of how an emulated human brain could. The problem is that human beings are potentially dangerous: they have volition, and they can enjoy destroying things. Basically they can be narcissistic psychopaths. All it would take would be a human being rich enough to have his mind uploaded in a machine, and then this man will get greedy for more computing power, even if to do so he would have to take over the World and extinct humanity in its organic form. Sounds very far-fetched, but as Hawking puts it: there is nothing in the physical laws that prevents a man-made device to perform as well or better than a human brain.
- ithkuil 12y ago
  Interesting read on the subject of what it's like to upload your mind to a device performing a simulation of your brain: http://en.wikipedia.org/wiki/Permutation_City http://en.wikipedia.org/wiki/Permutation_City
- T-A 12y ago
  The computational requirements are overwhelming though. There is a short discussion of this in section VIII of http://www.fqxi.org/community/forum/topic/2077 http://www.fqxi.org/community/forum/topic/2077 (click the PDF link); the currently top-ranked supercomputer turns out to be roughly five million times too slow to simulate a human brain in real time.
  - threatofrain 12y ago
    This might be a retarded question, but a simulation of a thing must always run more expensively / slower than the reality of the thing, right? And couldn't they build an organic material AI with inorganic interfaces for external manipulation?
monotypical 12y ago
Link to the actual article: http://www.independent.co.uk/news/science/stephen-hawking-transcendence-looks-at-the-implications-of-artificial-intelligence--but-are-we-taking-ai-seriously-enough-9313474.html http://www.independent.co.uk/news/science/stephen-hawking-tr...
beejiu 12y ago
I think the biggest problem as far as AI and automation is concerned is, what is left for us humans to do? At the moment, we work 42 hours per week (UK full time average), with many millions of people unemployed. Ultimately, our goal as a species, should be to build our infrastructure to be as efficient as possible. In an ideal world, we wouldn't have to go to work for 42 hours per week and many wouldn't need to work at all. We'd just enjoy ourselves 24/7. I think this shift from the 'work hard mantra' to an efficient world where this is much less 'manual' work to do (that includes many office jobs), is one that society, government and business underestimates.
gitaarik 12y ago
Bill Joy's article Why The Future Doesn't Need Us comes to mind: http://archive.wired.com/wired/archive/8.04/joy.html http://archive.wired.com/wired/archive/8.04/joy.html
duckingtest 12y ago
Intelligence is not magical. What matters is algorithmic complexity. At some point only throwing more computing power can help. Today, yes, maybe we could build an ai with close to human level intelligence, running on a very expensive super-computer(s) eating enormous amounts of energy. So what? Human-level ai isn't going to figure anything that humans can't figure out themselves, so it's not going to design new almost-magical ways to enhance itself. Especially because the entire human race is basically one giant parallel data processing system, although communication leaves much to be desired. Moore law isn't going to help - silicon technology is very close to stagnation [1]. I don't think singularity is possible until potential ai-accessible artificial computation capability is at least several orders of magnitude higher than human brain's, which is impossible with silicon-based technology. Nothing to worry about. [1] http://www.extremetech.com/computing/178529-this-is-what-the-death-of-moores-law-looks-like-euv-paused-indefinitely-450mm-wafers-halted-and-no-path-beyond-14nm http://www.extremetech.com/computing/178529-this-is-what-the...
richardw 12y ago
One thing I look forward to is greater logical input into high-level decision-making. Emotions, egos, racism, discrimination, nationalism, aggression - all these are things that we just don't need any more. I do think we need exceptional care when working on such a system but I also think the human mind of 50000 years ago leaves a lot to be desired when running in the 21st century. A mind that is ultra-rational would be a good adviser, helping to point out our blind spots and guide us to better thinking. We shouldn't have to learn from all mistakes every generation. (I'm certainly equally wary of blind acceptance of whatever science and technology can create. A favourite article of mine is http://archive.wired.com/wired/archive/8.04/joy.html http://archive.wired.com/wired/archive/8.04/joy.html - has some very serious warnings and consequences, even about my abovementioned thoughts.)
nroose 12y ago
I have a great deal of respect for Hawking, but... Lately it seems like he has said a couple of crazy things. We (as in most humans) are NOT going to get off the earth. Earth will always be the best chance for a planet for humans. Let's take care of it. And AI is NOT anywhere near becoming really intelligent. Anyone who has interacted with Siri knows that she has no real understanding. And the Google car and other autonomous driving systems are already safer and probably better for traffic than almost all drivers, but they still, and will most likely for a long time, need a human driver to take over under circumstances that actually require thought. AI is advancing, but it has always advanced slower than predicted. A lot slower than predicted. And while there are signs that AI will continue to be more and more useful as time goes on, and in some circumstances like recognizing driving, captchas and playing chess and jeopardy it will outperform humans in simple tests, it is not anywhere near, nor will it be for a long time, outperform the human brain in a general way. And certainly it is NOT similar to an alien (as in from outer space, not from another country on earth) civilization.

66 points by jaequery 12y ago

I share Stephen Hawking's concerns. I think AI is already at a point where maybe 10 or even 15 years down the road, artificial intelligence will be everywhere. Autonomous public transport, taxis, driver-less vehicles, autonomous crime prevention systems. The military is said to always be at least 10 years ahead of public tech research and development, look what they've already got now, imagine another 10 years. We've become a society that is too big and intelligent to fail, but that ignorance of being too smart and big to fail could be that our downfall is.

sidcool 12y ago
I share the concern partially. It's similar to other technologies which have downsides. Nuclear power for example. AI will have the power to help humanity prosper or cause its downfall. The balance will be for us to decide, again just like Nuclear power; for with great power comes great responsibility.

owens99 12y ago

Anyone know of a larger discussion related to this issue?

timClicks 12y ago
The Less Wrong community, various subreddits (/r/singularity, etc). Good terms to search for are "x-risk", "Jaan Tallinn" and "Nick Bostrom".

deleted 12y ago

[deleted]

eurleif 12y ago
Somehow, in the context of Stephen Hawking, this joke seems a bit insensitive.
- deleted 12y ago
  [deleted]

deleted 12y ago

[deleted]

deleted 12y ago

[deleted]

bad_alloc 12y ago

I recommend reading "Shadows of the Mind" by Roger Penrose. In that book he argues that strong artificial intelligence isn't achievable on classical computers, since the human brain seems to have the ability to do quantum computing in certain molecular structures. These processes probably can't be simulated efectively. So AI in today's computers probably won't work at all (although quantum computers or artificial neural networks might).

deleted 12y ago
[deleted]
dlss 12y ago
This has been discredited (along with Penrose's incompleteness theorem based "AI is impossible" conjecture before it) http://en.wikipedia.org/wiki/Quantum_mind#Roger_Penrose_and_Stuart_Hameroff http://en.wikipedia.org/wiki/Quantum_mind#Roger_Penrose_and_...
- ronaldx 12y ago
  Although specific claims might have been discredited, quantum processes are relevant to biology, e.g. http://www.nature.com/nature/journal/v446/n7137/abs/nature05678.html http://www.nature.com/nature/journal/v446/n7137/abs/nature05...
  - dlss 12y ago
    This is true, and it's also true that such quantum processes exist in the brain. Penrose's claim was that those quantum processes are a significant part of the computation that is consciousness. It was discredited by getting better upper bounds on how much quantum computation the brain is capable of (ie very little).

harshreality 12y ago

This is what MIRI (Machine Intelligence Research Institute) is investigating. http://intelligence.org/research/ http://intelligence.org/research/ The scariest part of the problem to me is this: http://yudkowsky.net/singularity/aibox http://yudkowsky.net/singularity/aibox If an AI can't be contained, then there has to be some failsafe that prevents any self-modifications which could turn it evil OR remove the failsafe. Or you have to believe that you can neutralize an evil AI before it destroys humanity.

deutronium 12y ago
Regarding the AI box experiment, I'm somewhat dubious about it, considering there doesn't seem to be any transcripts from when the AI 'escapes'. Which makes me think they use blackmail or something similar in the transcript.
- danbruc 12y ago
  I couldn't imagine how the AI could escape an hour ago, but after thinking about it for a while I am more and more convinced that it is actually hard to contain the AI. Containing the AI is like imprisoning a human for no reason and this will usually conflict with the gatekeepers moral values. Just imagine talking to human suffering in a tiny prison cell without having done anything wrong instead of the trapped AI. Can you see how it would be the right thing to open the cell's door? You are cruel not opening the door.
- jimrandomh 12y ago
  The AI experiment reproduced by Tuxedage: http://lesswrong.com/lw/gej/i_attempted_the_ai_box_experiment_and_lost/ http://lesswrong.com/lw/gej/i_attempted_the_ai_box_experimen... . 3 wins and 3 losses.
Udo 12y ago
The AIBox link contains a lot of buildup but ultimately delivers nothing. All we get is two people saying after a while "I let it out", questions as to the details of what happened are just not answered. I'm not denying it could be a fun game to play, but the article makes it way too painful and timeconsuming to find out what you're reading is in fact just the description of a game, and not an example for a strategy that could be used to let the AI out. That being said, yes, what's the point of having an AGI if you're not letting it "out"? It will probably escape anyway, so containment is not a solution. By the way, there is a lot of precedent in human history about the effectiveness of containing "dangerous" ideas by imprisoning the people who developed them. It generally doesn't work out. If you're developing AGI, you've pretty much already resigned to the idea that the world will change. The destruction of the human race is only one extreme outcome. Especially if the takeoff isn't very hard, there are a lot of scenarios where a net-beneficial AI could be developed. And all the engineering experience we have does indeed point to a very soft takeoff. Ultimately, I think the goal is NOT to generate a slave race with super powers. Yet that's what a lot of people are talking about. It's not only morally wrong but also self-evidently stupid. A lot of influential futurists are fans of that idea though, including Kurzweil. Useful AGI scenarios range from mutually beneficial coexistence to outright merging with the AGIs, and I think that's a goal well worth pursuing. Another strategy would be to keep augmenting ourselves and to continue on a transhumanization path until we are the AGI. You could indeed make an argument that's what's already happening.

daemonk 12y ago

What would the AI's motivation be? Will it understand what existence means and have the "desire" to survive? Is it possible that a self aware AI has already occurred in some form many times in the past, but since it doesn't have the anthropomorphic trait of wanting to exist, it never actively persisted? So what we are really afraid of is developing an AI with human motivations. But do the intelligence we want to artificially develop require human motivations? Aren't they two mutually exclusive traits? Maybe part of the fear is how we train the AI. If we are to imbue it with the total knowledge of our history, then maybe it will become more "human" and have those motivations. Is it possible to train it with hard data that aren't anthropomorphized in any way?

marvin 12y ago
An AI's motivation would be whatever it is programmed to do. Which is why it is so important to get this right the first time. Human motivations are incredibly complex, human moral systems have not yet been formalized and any attempt at doing this breaks down at the edge cases. This is not by any means a solved question, but there have been quite a few publications on this already. There are a hundred subtle errors of reasoning that ruin naïve and anthropocentric reasoning around morality and AI safety. Have a look at the lesswrong blog, for instance. Lots of interesting reading.
- sampo 12y ago
  > "An AI's motivation would be whatever it is programmed to do." Well we don't know whether it is possible to build systems as complex as human brain, and at the same time give them some specific goals.
- jeremiah256 12y ago
  If human motivations can be quantified and programmed into an AI system, my guess is 'loyalty' or 'patriotism' will be one of the first. "It is lamentable, that to be a good patriot one must become the enemy of the rest of mankind." Voltaire
andreyf 12y ago
I find it a rather convincing idea that human motivations are embodied in a sense that would take a lot of effort and no discernible purpose to replicate in an artificial system. I think more interesting is what Hawkins dismisses as "short term implications": who will control the intelligence enhancing software that we will see within our lifetimes and for what purpose? One example: what's up with these marketing+tech giants and what is their end-game? How accurately will we be able model desires/fears/motivations and how well can those with access to those models guide a person's beliefs and desires?

javindo 12y ago

I don't really want to come across as hostile to the views of such a well respect physicist, but this article represents the view which is often held by those who are very intelligent yet lack the actual study of AI. Anybody who has actually, or is currently, studying AI will know that there are fundamental differences in programming "AI" and some self aware magic computation which is almost entirely unfathomable. With current knowledge, at least, the only way to have any sort of learning is to design and implement algorithms to do so. There can of course be incredibly abstract knowledge base designs and the computation can assimilate incredibly complex knowledge from those, however it is almost a logical fallacy to suggest that we would ever lose control of something autonomous due to some sort of "rogue agent". Of course, there are very real risks with something like AI, but it's much less sinister than what the article suggests. For example, faulty code or errors in the knowledge base could lead it to make a bad decision, but humans can also do that. I don't think we should be dismissing the risks necessarily, but I do believe that it is completely far fetched to say we would be making the worst mistake in history if we are to dismiss a film as science fiction!

marvin 12y ago
On my side, I am surprised that so many AI researchers are unable to take the long-term view in this discussion. It's presumably because AI in the laboratory is still relatively primitive. No one is talking about magic. The human brain is not magic, neither is that of chimpanzees, rats, dolphins or gorillas. Intelligence is a purely physical phenomenon, which means it can be emulated by computers. Natural brains are also a product of evolution, which means (1) the development happens very slowly, (2) development is directed only towards evolutionary success and (3) there is no flexibility in how the thinking organs are constructed. Computer intelligence does not in principle have these limitations. It would be terribly anthropocentric to believe that humans are the most sophisticated intelligent entity that can exist in the physical world - after all, we are as far as we know the first such entity to emerge, so from our perspective the evolution of intelligence has now stopped. That's the feasability argument. The risk argument is that the consequences of an independent, runaway intelligent entity significantly more capable than humans would have such devastating consequences for humanity's future that even a small risk merits a significant effort to map out the territory. Respected scientists have said "it is impossible" to hundreds of things that proved to be quite simple, so this is not an argument. Even if you don't buy the previous part of my reasoning, there is still risk here. In principle, there are any number of things that could preclude advanced AI in the near future even if the above reasoning is correct (too difficult, requires too much computing power, uses different computational techniques), but seeing as we don't know the unknowns here, taking the cautious route is the correct thing to do. This has been a scientific principle for decades, no reason to drop now. Do AI researchers have any arguments opposing this that don't amount to "the AI we have created up until now is not very good"?

nanomage 12y ago

so, Mr. Hawking hasseen 'The Animetrix,' I gather?

tim333 12y ago

Assuming strong AI arises, which I think likely, and given human nature I imagine people will build all sorts of AI's, 'evil' and 'good', though those terms are a bit subjective. Probably the likes of Kim Jong Un would love to have an army of self reproducing killer robots and Google will try to build nice ones and the US defence department will continue with drone like things. I agree with Hawking it could be a problem and hope the nice ones win out as they probably will. Fingers crossed.

tmikaeld 12y ago

Simplistically: If Win = Continue If Loose = Change Basic evolutionary principle. Then giving the tools of doing, would be required to consume the formula. I argue that an AI with limitations would not work.

grondilu 12y ago

Even if AI is harmless, there is still the risk of brain emulation. To me this is a greater threat, because if we don't know how a fully artificial mind would behave, we can get an idea of how an emulated human brain could. The problem is that human beings are potentially dangerous: they have volition, and they can enjoy destroying things. Basically they can be narcissistic psychopaths. All it would take would be a human being rich enough to have his mind uploaded in a machine, and then this man will get greedy for more computing power, even if to do so he would have to take over the World and extinct humanity in its organic form. Sounds very far-fetched, but as Hawking puts it: there is nothing in the physical laws that prevents a man-made device to perform as well or better than a human brain.

ithkuil 12y ago
Interesting read on the subject of what it's like to upload your mind to a device performing a simulation of your brain: http://en.wikipedia.org/wiki/Permutation_City http://en.wikipedia.org/wiki/Permutation_City
T-A 12y ago
The computational requirements are overwhelming though. There is a short discussion of this in section VIII of http://www.fqxi.org/community/forum/topic/2077 http://www.fqxi.org/community/forum/topic/2077 (click the PDF link); the currently top-ranked supercomputer turns out to be roughly five million times too slow to simulate a human brain in real time.
- threatofrain 12y ago
  This might be a retarded question, but a simulation of a thing must always run more expensively / slower than the reality of the thing, right? And couldn't they build an organic material AI with inorganic interfaces for external manipulation?

monotypical 12y ago

Link to the actual article: http://www.independent.co.uk/news/science/stephen-hawking-transcendence-looks-at-the-implications-of-artificial-intelligence--but-are-we-taking-ai-seriously-enough-9313474.html http://www.independent.co.uk/news/science/stephen-hawking-tr...

beejiu 12y ago

I think the biggest problem as far as AI and automation is concerned is, what is left for us humans to do? At the moment, we work 42 hours per week (UK full time average), with many millions of people unemployed. Ultimately, our goal as a species, should be to build our infrastructure to be as efficient as possible. In an ideal world, we wouldn't have to go to work for 42 hours per week and many wouldn't need to work at all. We'd just enjoy ourselves 24/7. I think this shift from the 'work hard mantra' to an efficient world where this is much less 'manual' work to do (that includes many office jobs), is one that society, government and business underestimates.

gitaarik 12y ago

Bill Joy's article Why The Future Doesn't Need Us comes to mind: http://archive.wired.com/wired/archive/8.04/joy.html http://archive.wired.com/wired/archive/8.04/joy.html

duckingtest 12y ago

Intelligence is not magical. What matters is algorithmic complexity. At some point only throwing more computing power can help. Today, yes, maybe we could build an ai with close to human level intelligence, running on a very expensive super-computer(s) eating enormous amounts of energy. So what? Human-level ai isn't going to figure anything that humans can't figure out themselves, so it's not going to design new almost-magical ways to enhance itself. Especially because the entire human race is basically one giant parallel data processing system, although communication leaves much to be desired. Moore law isn't going to help - silicon technology is very close to stagnation [1]. I don't think singularity is possible until potential ai-accessible artificial computation capability is at least several orders of magnitude higher than human brain's, which is impossible with silicon-based technology. Nothing to worry about. [1] http://www.extremetech.com/computing/178529-this-is-what-the-death-of-moores-law-looks-like-euv-paused-indefinitely-450mm-wafers-halted-and-no-path-beyond-14nm http://www.extremetech.com/computing/178529-this-is-what-the...

richardw 12y ago

One thing I look forward to is greater logical input into high-level decision-making. Emotions, egos, racism, discrimination, nationalism, aggression - all these are things that we just don't need any more. I do think we need exceptional care when working on such a system but I also think the human mind of 50000 years ago leaves a lot to be desired when running in the 21st century. A mind that is ultra-rational would be a good adviser, helping to point out our blind spots and guide us to better thinking. We shouldn't have to learn from all mistakes every generation. (I'm certainly equally wary of blind acceptance of whatever science and technology can create. A favourite article of mine is http://archive.wired.com/wired/archive/8.04/joy.html http://archive.wired.com/wired/archive/8.04/joy.html - has some very serious warnings and consequences, even about my abovementioned thoughts.)

nroose 12y ago

I have a great deal of respect for Hawking, but... Lately it seems like he has said a couple of crazy things. We (as in most humans) are NOT going to get off the earth. Earth will always be the best chance for a planet for humans. Let's take care of it. And AI is NOT anywhere near becoming really intelligent. Anyone who has interacted with Siri knows that she has no real understanding. And the Google car and other autonomous driving systems are already safer and probably better for traffic than almost all drivers, but they still, and will most likely for a long time, need a human driver to take over under circumstances that actually require thought. AI is advancing, but it has always advanced slower than predicted. A lot slower than predicted. And while there are signs that AI will continue to be more and more useful as time goes on, and in some circumstances like recognizing driving, captchas and playing chess and jeopardy it will outperform humans in simple tests, it is not anywhere near, nor will it be for a long time, outperform the human brain in a general way. And certainly it is NOT similar to an alien (as in from outer space, not from another country on earth) civilization.