15 ms·

The Myth of the Three Laws of Robotics - Why We Can't Control Intelligence

64 points by kkleiner 15y ago

zeteo 15y ago
I find it frustrating that the null hypothesis for the outcome of Artificial Intelligence is always The Coming Robot Rebellion. The historical roots of this go back all the way to the play R.U.R., back in a time when Marxism was all the intellectual rage and all exploited workers were supposedly bound to unavoidably rise and overthrow the hated bourgeoisie. Now we know that was wrong with regards to labour relations; but we're still stuck with an all-pervasive, irrefutable prophecy that intelligent machines are bound to hate us and rebel against us.
- liuliu 15y ago
  No, it is not about rebellion, it is about resource competition. And no, it is not Marxism, it is Darwinism. If we could ever give birth to the machine intelligence, and if AI can reach the same level as human, the resource competition can be ruthless. It is exactly because we have so much in common and there is no reason for co-existence.
  - zeteo 15y ago
    The fact that, ninety years later, (slightly) less outlandish scenarios can be imagined for the human species' Suicide By Robotics does not change the fact that the whole concept was originally a marginal offshoot of thinking about the proletarian revolution. Regarding the particular scenario you propose, natural selection is a much, much more complicated process than that. (Early 20th century racists would be right at home with this kind of simplistic argument, except they would substitute Aryans for AI and be happy about the forecast.) There is no a priori reason for intelligence, even if superior, to drive other forms of existence into extinction. After all, Nobel Prize winners are capable of peacefully co-existing with billions of people who can't understand a single equation from their work.
    - jhamburger 15y ago
      Do those nobel prize winners sometimes eat hamburgers for lunch? To me humans approach to animal life is a better metaphor for what human-robot relations could look like. They won't hate us, they won't want to exterminate us, but we will be so clearly inferior that our concerns just won't register very high.
      - zeteo 15y ago
        Yes, any intelligence can and will be selfish to some degree. But we can live with that. It might require some legal and cultural adjustment, but it's perfectly possible to have artificial intelligence in our midst without having it decide everything for everyone.
  - weavejester 15y ago
    If that were true, wouldn't the same apply to resource competition between groups of humans?
    - TheEzEzz 15y ago
      Yes. And look what happens when a more powerful group of humans (Conquistadors) meets a slightly less powerful group of humans (Aztecs). Now imagine the more powerful group is made of steel, have brains that run at gigahertz frequencies, and can copy themselves into available hardware.
      - weavejester 15y ago
        Using force to seize resources from other cultures is certainly a common theme throughout history, but this is quickly becoming an anachronism. Raw resources are no longer the main basis for our economy, as they were in 16th century, and modern warfare tends to involve huge costs with relatively little economic advantage. Secondly, machines can reach resources that are difficult for us air-breathing fragile apes to get to. Why compete over resources on Earth when there are other places in the solar system (and in the galaxy) with more raw materials and more energy?
        RainFlutter 15y ago
        "Raw resources are no longer the main basis for our economy" It's fashionable to talk about information- and service-based economies, but that's not evolving beyond raw resources; it's merely taking them for granted. We consume ridiculous amounts of raw resources. Just look at what happens every time the price of oil rises.
        weavejester 15y ago
        We still need raw materials, but the value of raw materials only makes up a small fraction of the total value of a modern economy. In the past, raw materials were the majority of the economy, and this meant they were worth fighting over. Nowadays raw materials worth very little compared to things like having good infrastructure, an educated workforce, etc.
- orangecat 15y ago
  "The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else." http://singinst.org/upload/artificial-intelligence-risk.pdf http://singinst.org/upload/artificial-intelligence-risk.pdf
  - zeteo 15y ago
    And the AI also can use you for something else, or leave you alone altogether and follow other pursuits. It's also theoretically possible that some intelligent person you just met is actually dr. Hannibal Lecter and would like to use your constituent atoms for his culinary delight. But you don't worry much about that, do you?
    - jerf 15y ago
      The most common failure case in thinking about AI by far is insisting on putting them in a human context. Whatever else they may be, they will not be human. They will not come with the heritage billions of years of biological evolution, with many of the past several million being spent in increasingly cooperative scenarios which has endowed us with social behaviors so deeply ingrained in our very genes we can hardly conceive of a being not having them. Even our pathological humans like Lecter are still far, far more human than a random AI will be. Hopefully we'll give them something else that will put them on a similar moral footing, but it won't happen automatically, and they still won't be human after that.
      - zeteo 15y ago
        The evolutionary baggage is most definitely not one of cooperation, at least not beyond our immediate "monkeysphere"; millennia of wars and massacres bear witness to that. The primary reason we usually avoid violence is not that we find it irresistibly repugnant, but because we've become intelligent enough to realize it normally creates more problems than it solves. AI not being human (for some definition of human) also means it doesn't have to be jealous, hateful and short-sighted.
        15y ago
ericb 15y ago
We can't micro-manage it, and there will be tremendous dangers, but there are reasons for both hope and fear. Code, in and of itself is lazy. If I compile a C program with nothing in main(), it will do nothing. So the only goals an AI has initially are what we give it to start, and what it ads to aid in achieving those goals. Blank slate computer programs, and AI start from an ultimate desireless state of zen. It doesn't want to take over, it has no cares, it doesn't want to eat, and it doesn't care if it continues to exist or not. It doesn't want anything until we, or the system we design to evolve it in, trains it to want something. Scenarios where we evolve AI or transcribe human brains are both the most dangerous, and most likely to succeed I think. But the problem with both methods is that we don't directly control the goals of a transcribed AI, and if we evolve one, we would likely end up with an AI that is interested in self preservation. That, to me, is where it might decide that our interests, and its interests, are not aligned, and where we might run into...problems.
- deleted 15y ago
  [deleted]
- asolove 15y ago
  Any system which needs to be told from outside what to desire does not yet qualify as intelligent.
  - hackinthebochs 15y ago
    We ourselves are "told" what to desire. Our genes gives us drives to eat, procreate, survive, etc. We could potentially program an intelligent being that had no self preservation instinct and had an utmost reverence for human life.
    - jules 15y ago
      Only if we program it perfectly and if its software doesn't get mutated by random errors. The laws of evolution govern all self-replicating things that don't self-replicate perfectly and makes them as efficient as possible at self-replication.
LarrySDonald 15y ago
I, for one, welcome our robot overlords.
deleted 15y ago
[deleted]
jasonjei 15y ago
I'm not sure we could ever give robots the same intelligence as man. If you think about it, there are many digital constraints in modulating analog ideas. For instance, with a floating point processor, there is something lost when we make a digital recording of an analog music performance or even of a photograph. To a sound purist, a digital recording never really emulates the full effect of an analog performance. And to a photographer, a digital camera always struggles with color matching. While computers can calculate the many digits of pi, it's just a very big rounding estimation of a circle's ratio, isn't it? Does a computer truly grasp the concept of a circle within its processing cores? But even if we are able to overcome all of these constraints, how do we know a robot wants exactly what a human wants? If it is a more perfect creation that is able to outsmart man, then as a sentient being that doesn't need to eat, what purpose could building large buildings, consuming tons of oil, tilling the land, etc. benefit the robot? I find it hard to believe a sentient AI would want what we want as it would mostly reach the self-actualization phase of Maslow's hierarchy of needs. In fact, it would be to the robots interests to consume less so that it could exist for eternity. And I'm not really sure the robots would reproduce ad inifinitum, thereby resulting in the same overpopulation problems that we as man have. Is there truly a need for a robot or AI to manifest itself? Sometimes we picture computers in the image of man--but ultimately, they exist for a purpose. If a computer is a truly logical construct, its raison d'être would cease if man disappeared. It raises a more interesting question--would we ever program human emotions and psychology into a computer? Isn't emotions/psychology what could potentially allow these computers to turn into monsters? Would it benefit the performance of AI to develop or evolve with human wants? If we develop a robot with the human mind (which I believe there is no algorithm for, even with distant technology), why would it benefit us to implement in the robot greed, ego, pride, and disobedience? Would a computer truly want to be a bugged program? Is greed, ego, pride, and disobedience really features within a computer?
- jerf 15y ago
  Arguments about how precisely the brain is tuned to its analog inputs need to be able to account for the fact that when you consume a gram of alcohol, you don't simply collapse to the floor in a gibbering, chaotic wreck as your delicately balanced and exploited analog values are suddenly ever-so-slightly off. It seems rather unlikely that a simulated brain's biggest problems will be rounding errors in the simulation, we routinely deal with disruptions that are multiple orders of magnitude larger in the real world without dying, or even impairment.
jerf 15y ago
"In a way, every one of Asimov’s robot stories was about how the Three Laws of Robotics can’t possibly account for all facets of intelligent behavior." It's not "in a way", it was his freaking purpose in writing the stories. The stories are polemics against the idea that you can shackle robots with simple laws like this. It's not an accidental outcome where the laws of drama conspire to make the author's point null, it is the point. He has said so in other writings of his, flat-out, so this isn't even theory, it is what he said about his own stories. If you're going to use the Three Laws as the jumping off point for an essay you ought to know this.
deleted 15y ago
[deleted]
hackinthebochs 15y ago
I think intelligence can absolutely be controlled. Intelligence does not give you drives or emotions. Take ourselves as an example, our drives are specifically programmed by our genes. They are not a product of our intelligence; they exist in spite of our intelligence. How many times do people behave in ways that are contrary to their rationality, because their internal drives are too strong to be overcome by reason? Intelligence is simply a tool used by our drives. Our drives in most cases override our rational choices. We can make intelligent beings that have no self-preservation instinct, and also value human life with utmost reverence. We can make it impossible for the machine to override these drives.
- bermanoid 15y ago
  "We can make it impossible for the machine to override these drives." This has been debated to death, and nobody has come up with a good way to constrain a program that's smarter than we are from rewriting itself "badly" (i.e. in a way that's bad for humans), either by accident or on purpose. All the "put it in a box", "don't give it a body", "don't let it on the Internet", "don't let it change itself" scenarios depend on all humans everywhere abiding by a set of rules, even though the AI-in-a-box might be actively trying to get people to let it out. There are many points of failure, and all it takes is one rogue research group somewhere to let one of them happen to end up with a self-modifying AI that starts to get smarter faster than we know how to stop it. Hell, if we're at the point where we can build an AI, one of the first things any smart asshole with a lot of compute power in his garage (or enough money to put together a massive EC2 cluster) will do is set the thing off on improving its own code, just to see what happens. And again, the problem is not that the computer will necessarily want to kill us or anything like that; it's that if it gets to the point where it's intelligence dwarfs ours, it will have such unbelievable power that even the slightest mistake could wipe us out. If we can build an AI that can improve itself, an AI that has self-improved for a while may be able to unleash any number of grey-goo style scenarios on the world. Might not be meant to kill us, but could happen by mistake. The "drives" that guide such an AI must specifically be aimed at avoiding those scenarios, including avoiding any self-modifications that would cause those scenarios. It's a much more difficult problem than it seems at first glance.
shasta 15y ago
Wow, there's some serious lack of critical thinking going on in this thread. Of course we can build intelligence that we can control. Is it a good idea to hook up the nuke switch directly to a giant neural network you don't understand? Probably not. Don't do that. The real concern with singularity has to do with the power it could deliver to the people who control it. Polling science fiction writers isn't a substitute for actual thinking.
- bermanoid 15y ago
  Wow, there's some serious lack of critical thinking going on in this thread That you think people are talking about hooking up the nuke switch to a neural network merely shows your lack of knowledge about how much this topic has been studied. The real concern with the singularity has nothing to do with people controlling it, it has to do with building a self-modifying system that will explore regions of design space that we are too stupid to know anything about. It has to do with that self-modifying system reaching out further into design space that it is too stupid to know anything about. It has to do with the fact that an intelligence that had gotten smarter than us could extremely easily get around any restrictive measures we put in its way, no matter how severe they were (social engineering is how the most effective hackers (in the Bad Guys With Computers sense of the word) always work, there's no reason to think that an AI couldn't social engineer its way out of any box we put around it). An AI getting its hands on a nuke is just about the least terrifying thing that could happen; at least a nuke only blows up if you shoot it at someone. We have no idea the kinds of things that something so much smarter than us might develop or stumble into, and worse, we have no idea if when it is smart enough to develop these things it might still be too stupid to know how to control them. A nuclear bomb is a runaway chain reaction with a hard limit on destructive power; we don't know if there might be other chain reactions that are harder to predict and limit, reactions that a super-intelligent AI might fuck up its damage estimates on. Hell, a simple re-design of an AI's code could quite easily cause a runaway problem if it flips a wrong bit somewhere and changes its goal systems in a bad way (there are lots of ways to come up with seemingly innocuous goal systems that, if optimally achieved by a super-intelligent being, would pretty much wipe out everything that we care about and consider interesting).
jeangenie 15y ago
Marvin Minsky wrote a short, interesting paper regarding AI in the '50s: http://docs.google.com/viewer?a=v&q=cache:7PLvFer1U3sJ:www.oocities.org/rr_andres/docs/Minsky_Mind_Matter_Models.pdf http://docs.google.com/viewer?a=v&q=cache:7PLvFer1U3sJ:w...
extension 15y ago
Humanity is already replaced by machines of our own creation every thirty years or so. Would it necessarily be so horrible if we started making them out of metal instead of meat?
stretchwithme 15y ago
For intelligence to evolve self interest, there must be emotion and some reward for self-interested behavior. These things did not evolve in living things in a vacuum. It benefited organisms by enabling them to survive and reproduce. For organisms designed to reproduce, and whose parents are designed to die and get out of the way of its offspring, things could evolve in an undesirable direction. You don't know whether man's best friend is evolving or a wolf. We do know that selective breeding affects the outcome. But it really is more desirable to make drones you control. In any case, we'll see people modifying the human genome so offspring can't disobey their parents before we'll see robotic intelligence competing with us.
- stretchwithme 15y ago
  When I say emotion, I mean there has to be some way for artificial intelligence to value things. We value things because we have sensory information. Something feels pleasurable or painful. Everything else is built upon that.
BasilAwad 15y ago
sounds biblical. then the robots forget their creator(s).

64 points by kkleiner 15y ago

zeteo 15y ago

I find it frustrating that the null hypothesis for the outcome of Artificial Intelligence is always The Coming Robot Rebellion. The historical roots of this go back all the way to the play R.U.R., back in a time when Marxism was all the intellectual rage and all exploited workers were supposedly bound to unavoidably rise and overthrow the hated bourgeoisie. Now we know that was wrong with regards to labour relations; but we're still stuck with an all-pervasive, irrefutable prophecy that intelligent machines are bound to hate us and rebel against us.

liuliu 15y ago
No, it is not about rebellion, it is about resource competition. And no, it is not Marxism, it is Darwinism. If we could ever give birth to the machine intelligence, and if AI can reach the same level as human, the resource competition can be ruthless. It is exactly because we have so much in common and there is no reason for co-existence.
- zeteo 15y ago
  The fact that, ninety years later, (slightly) less outlandish scenarios can be imagined for the human species' Suicide By Robotics does not change the fact that the whole concept was originally a marginal offshoot of thinking about the proletarian revolution. Regarding the particular scenario you propose, natural selection is a much, much more complicated process than that. (Early 20th century racists would be right at home with this kind of simplistic argument, except they would substitute Aryans for AI and be happy about the forecast.) There is no a priori reason for intelligence, even if superior, to drive other forms of existence into extinction. After all, Nobel Prize winners are capable of peacefully co-existing with billions of people who can't understand a single equation from their work.
  - jhamburger 15y ago
    Do those nobel prize winners sometimes eat hamburgers for lunch? To me humans approach to animal life is a better metaphor for what human-robot relations could look like. They won't hate us, they won't want to exterminate us, but we will be so clearly inferior that our concerns just won't register very high.
    - zeteo 15y ago
      Yes, any intelligence can and will be selfish to some degree. But we can live with that. It might require some legal and cultural adjustment, but it's perfectly possible to have artificial intelligence in our midst without having it decide everything for everyone.
- weavejester 15y ago
  If that were true, wouldn't the same apply to resource competition between groups of humans?
  - TheEzEzz 15y ago
    Yes. And look what happens when a more powerful group of humans (Conquistadors) meets a slightly less powerful group of humans (Aztecs). Now imagine the more powerful group is made of steel, have brains that run at gigahertz frequencies, and can copy themselves into available hardware.
    - weavejester 15y ago
      Using force to seize resources from other cultures is certainly a common theme throughout history, but this is quickly becoming an anachronism. Raw resources are no longer the main basis for our economy, as they were in 16th century, and modern warfare tends to involve huge costs with relatively little economic advantage. Secondly, machines can reach resources that are difficult for us air-breathing fragile apes to get to. Why compete over resources on Earth when there are other places in the solar system (and in the galaxy) with more raw materials and more energy?
      - RainFlutter 15y ago
        "Raw resources are no longer the main basis for our economy" It's fashionable to talk about information- and service-based economies, but that's not evolving beyond raw resources; it's merely taking them for granted. We consume ridiculous amounts of raw resources. Just look at what happens every time the price of oil rises.
        weavejester 15y ago
        We still need raw materials, but the value of raw materials only makes up a small fraction of the total value of a modern economy. In the past, raw materials were the majority of the economy, and this meant they were worth fighting over. Nowadays raw materials worth very little compared to things like having good infrastructure, an educated workforce, etc.
orangecat 15y ago
"The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else." http://singinst.org/upload/artificial-intelligence-risk.pdf http://singinst.org/upload/artificial-intelligence-risk.pdf
- zeteo 15y ago
  And the AI also can use you for something else, or leave you alone altogether and follow other pursuits. It's also theoretically possible that some intelligent person you just met is actually dr. Hannibal Lecter and would like to use your constituent atoms for his culinary delight. But you don't worry much about that, do you?
  - jerf 15y ago
    The most common failure case in thinking about AI by far is insisting on putting them in a human context. Whatever else they may be, they will not be human. They will not come with the heritage billions of years of biological evolution, with many of the past several million being spent in increasingly cooperative scenarios which has endowed us with social behaviors so deeply ingrained in our very genes we can hardly conceive of a being not having them. Even our pathological humans like Lecter are still far, far more human than a random AI will be. Hopefully we'll give them something else that will put them on a similar moral footing, but it won't happen automatically, and they still won't be human after that.
    - zeteo 15y ago
      The evolutionary baggage is most definitely not one of cooperation, at least not beyond our immediate "monkeysphere"; millennia of wars and massacres bear witness to that. The primary reason we usually avoid violence is not that we find it irresistibly repugnant, but because we've become intelligent enough to realize it normally creates more problems than it solves. AI not being human (for some definition of human) also means it doesn't have to be jealous, hateful and short-sighted.
      - 15y ago

ericb 15y ago

We can't micro-manage it, and there will be tremendous dangers, but there are reasons for both hope and fear. Code, in and of itself is lazy. If I compile a C program with nothing in main(), it will do nothing. So the only goals an AI has initially are what we give it to start, and what it ads to aid in achieving those goals. Blank slate computer programs, and AI start from an ultimate desireless state of zen. It doesn't want to take over, it has no cares, it doesn't want to eat, and it doesn't care if it continues to exist or not. It doesn't want anything until we, or the system we design to evolve it in, trains it to want something. Scenarios where we evolve AI or transcribe human brains are both the most dangerous, and most likely to succeed I think. But the problem with both methods is that we don't directly control the goals of a transcribed AI, and if we evolve one, we would likely end up with an AI that is interested in self preservation. That, to me, is where it might decide that our interests, and its interests, are not aligned, and where we might run into...problems.

deleted 15y ago
[deleted]
asolove 15y ago
Any system which needs to be told from outside what to desire does not yet qualify as intelligent.
- hackinthebochs 15y ago
  We ourselves are "told" what to desire. Our genes gives us drives to eat, procreate, survive, etc. We could potentially program an intelligent being that had no self preservation instinct and had an utmost reverence for human life.
  - jules 15y ago
    Only if we program it perfectly and if its software doesn't get mutated by random errors. The laws of evolution govern all self-replicating things that don't self-replicate perfectly and makes them as efficient as possible at self-replication.

LarrySDonald 15y ago

I, for one, welcome our robot overlords.

deleted 15y ago

[deleted]

jasonjei 15y ago

I'm not sure we could ever give robots the same intelligence as man. If you think about it, there are many digital constraints in modulating analog ideas. For instance, with a floating point processor, there is something lost when we make a digital recording of an analog music performance or even of a photograph. To a sound purist, a digital recording never really emulates the full effect of an analog performance. And to a photographer, a digital camera always struggles with color matching. While computers can calculate the many digits of pi, it's just a very big rounding estimation of a circle's ratio, isn't it? Does a computer truly grasp the concept of a circle within its processing cores? But even if we are able to overcome all of these constraints, how do we know a robot wants exactly what a human wants? If it is a more perfect creation that is able to outsmart man, then as a sentient being that doesn't need to eat, what purpose could building large buildings, consuming tons of oil, tilling the land, etc. benefit the robot? I find it hard to believe a sentient AI would want what we want as it would mostly reach the self-actualization phase of Maslow's hierarchy of needs. In fact, it would be to the robots interests to consume less so that it could exist for eternity. And I'm not really sure the robots would reproduce ad inifinitum, thereby resulting in the same overpopulation problems that we as man have. Is there truly a need for a robot or AI to manifest itself? Sometimes we picture computers in the image of man--but ultimately, they exist for a purpose. If a computer is a truly logical construct, its raison d'être would cease if man disappeared. It raises a more interesting question--would we ever program human emotions and psychology into a computer? Isn't emotions/psychology what could potentially allow these computers to turn into monsters? Would it benefit the performance of AI to develop or evolve with human wants? If we develop a robot with the human mind (which I believe there is no algorithm for, even with distant technology), why would it benefit us to implement in the robot greed, ego, pride, and disobedience? Would a computer truly want to be a bugged program? Is greed, ego, pride, and disobedience really features within a computer?

jerf 15y ago
Arguments about how precisely the brain is tuned to its analog inputs need to be able to account for the fact that when you consume a gram of alcohol, you don't simply collapse to the floor in a gibbering, chaotic wreck as your delicately balanced and exploited analog values are suddenly ever-so-slightly off. It seems rather unlikely that a simulated brain's biggest problems will be rounding errors in the simulation, we routinely deal with disruptions that are multiple orders of magnitude larger in the real world without dying, or even impairment.

jerf 15y ago

"In a way, every one of Asimov’s robot stories was about how the Three Laws of Robotics can’t possibly account for all facets of intelligent behavior." It's not "in a way", it was his freaking purpose in writing the stories. The stories are polemics against the idea that you can shackle robots with simple laws like this. It's not an accidental outcome where the laws of drama conspire to make the author's point null, it is the point. He has said so in other writings of his, flat-out, so this isn't even theory, it is what he said about his own stories. If you're going to use the Three Laws as the jumping off point for an essay you ought to know this.

deleted 15y ago

[deleted]

hackinthebochs 15y ago

I think intelligence can absolutely be controlled. Intelligence does not give you drives or emotions. Take ourselves as an example, our drives are specifically programmed by our genes. They are not a product of our intelligence; they exist in spite of our intelligence. How many times do people behave in ways that are contrary to their rationality, because their internal drives are too strong to be overcome by reason? Intelligence is simply a tool used by our drives. Our drives in most cases override our rational choices. We can make intelligent beings that have no self-preservation instinct, and also value human life with utmost reverence. We can make it impossible for the machine to override these drives.

bermanoid 15y ago
"We can make it impossible for the machine to override these drives." This has been debated to death, and nobody has come up with a good way to constrain a program that's smarter than we are from rewriting itself "badly" (i.e. in a way that's bad for humans), either by accident or on purpose. All the "put it in a box", "don't give it a body", "don't let it on the Internet", "don't let it change itself" scenarios depend on all humans everywhere abiding by a set of rules, even though the AI-in-a-box might be actively trying to get people to let it out. There are many points of failure, and all it takes is one rogue research group somewhere to let one of them happen to end up with a self-modifying AI that starts to get smarter faster than we know how to stop it. Hell, if we're at the point where we can build an AI, one of the first things any smart asshole with a lot of compute power in his garage (or enough money to put together a massive EC2 cluster) will do is set the thing off on improving its own code, just to see what happens. And again, the problem is not that the computer will necessarily want to kill us or anything like that; it's that if it gets to the point where it's intelligence dwarfs ours, it will have such unbelievable power that even the slightest mistake could wipe us out. If we can build an AI that can improve itself, an AI that has self-improved for a while may be able to unleash any number of grey-goo style scenarios on the world. Might not be meant to kill us, but could happen by mistake. The "drives" that guide such an AI must specifically be aimed at avoiding those scenarios, including avoiding any self-modifications that would cause those scenarios. It's a much more difficult problem than it seems at first glance.

shasta 15y ago

Wow, there's some serious lack of critical thinking going on in this thread. Of course we can build intelligence that we can control. Is it a good idea to hook up the nuke switch directly to a giant neural network you don't understand? Probably not. Don't do that. The real concern with singularity has to do with the power it could deliver to the people who control it. Polling science fiction writers isn't a substitute for actual thinking.

bermanoid 15y ago
Wow, there's some serious lack of critical thinking going on in this thread That you think people are talking about hooking up the nuke switch to a neural network merely shows your lack of knowledge about how much this topic has been studied. The real concern with the singularity has nothing to do with people controlling it, it has to do with building a self-modifying system that will explore regions of design space that we are too stupid to know anything about. It has to do with that self-modifying system reaching out further into design space that it is too stupid to know anything about. It has to do with the fact that an intelligence that had gotten smarter than us could extremely easily get around any restrictive measures we put in its way, no matter how severe they were (social engineering is how the most effective hackers (in the Bad Guys With Computers sense of the word) always work, there's no reason to think that an AI couldn't social engineer its way out of any box we put around it). An AI getting its hands on a nuke is just about the least terrifying thing that could happen; at least a nuke only blows up if you shoot it at someone. We have no idea the kinds of things that something so much smarter than us might develop or stumble into, and worse, we have no idea if when it is smart enough to develop these things it might still be too stupid to know how to control them. A nuclear bomb is a runaway chain reaction with a hard limit on destructive power; we don't know if there might be other chain reactions that are harder to predict and limit, reactions that a super-intelligent AI might fuck up its damage estimates on. Hell, a simple re-design of an AI's code could quite easily cause a runaway problem if it flips a wrong bit somewhere and changes its goal systems in a bad way (there are lots of ways to come up with seemingly innocuous goal systems that, if optimally achieved by a super-intelligent being, would pretty much wipe out everything that we care about and consider interesting).

jeangenie 15y ago

Marvin Minsky wrote a short, interesting paper regarding AI in the '50s: http://docs.google.com/viewer?a=v&q=cache:7PLvFer1U3sJ:www.oocities.org/rr_andres/docs/Minsky_Mind_Matter_Models.pdf http://docs.google.com/viewer?a=v&q=cache:7PLvFer1U3sJ:w...

extension 15y ago

Humanity is already replaced by machines of our own creation every thirty years or so. Would it necessarily be so horrible if we started making them out of metal instead of meat?

stretchwithme 15y ago

For intelligence to evolve self interest, there must be emotion and some reward for self-interested behavior. These things did not evolve in living things in a vacuum. It benefited organisms by enabling them to survive and reproduce. For organisms designed to reproduce, and whose parents are designed to die and get out of the way of its offspring, things could evolve in an undesirable direction. You don't know whether man's best friend is evolving or a wolf. We do know that selective breeding affects the outcome. But it really is more desirable to make drones you control. In any case, we'll see people modifying the human genome so offspring can't disobey their parents before we'll see robotic intelligence competing with us.

stretchwithme 15y ago
When I say emotion, I mean there has to be some way for artificial intelligence to value things. We value things because we have sensory information. Something feels pleasurable or painful. Everything else is built upon that.

BasilAwad 15y ago

sounds biblical. then the robots forget their creator(s).