6 ms·

Do we understand ethics well enough to build “Friendly artificial intelligence”?

42 points by pldpld 15y ago

vessenes 15y ago
Best suggestion of the article is that we scorn AI researchers who do not have a credible claim that their designs will maintain a basic agreed-on value system after a billion self-managed iterations and upgrades by the AI. This is a fascinating and broad-ranging criticism of AI, and it's interesting to me because the author is clearly considering 'what happens if we are successful?'. Definitely worth a read.
- gnosis 15y ago
  "Best suggestion of the article is that we scorn AI researchers who do not have a credible claim that their designs will maintain a basic agreed-on value system after a billion self-managed iterations and upgrades by the AI." Unfortunately, I don't think that's really going to happen. When people refuse to fund research in a promising area, others wind up taking up the slack. Witness America's prohibition on fetal stem-cell research. The US lost its lead in that field when researchers from other countries continued the work. The same will be much more true of AI research, as a working strong AI would probably be seen as a goose that lays golden eggs. Of course, it's also a Pandora's Box. So we should be careful. But I don't think scorning researchers in the field is going to be very effective.
- _delirium 15y ago
  That's a pretty high bar that would be pretty controversial if you applied it to other fields. For example, should we scorn genetics researchers who do not have a credible claim that their organisms will remain harmless after a billion generations of evolution and recombination? That's more or less what many of the anti-GMO arguments boil down to, that we ought to require genetically-modified organisms to be provably safe, both as they exist now, and in all possible future ways they could evolve and interact with other organisms. (And since that bar is very hard to reach, therefore, their arguments go, we should be careful about funding such research to begin with, and definitely shouldn't let any of its results out into the wild, e.g. into crops.) If anything, the argument there is stronger, because evolving biological organisms that can pose a threat to humans actually exist, whereas evolving machines that can pose a threat to humans are sci-fi, and likely to remain so for a very long time. Why regulate the latter one more stringently?
  - yummyfajitas 15y ago
    The search space of biological organisms been well explored. It's unlikely that one will develop which is vastly more dangerous than the ones which already exist. The same cannot be said for robots/AI.
    - jessriedel 15y ago
      > The search space of biological organisms been well explored. Not at all true. The space of possible biological organisms is searched in a highly nonuniform manner by evolution, and the human search strategy is fundamentally different. It's overwhelmingly likely that there competitive human-constructible organisms which could never be produced by evolution in the past 4 billion years.
      - yummyfajitas 15y ago
        This is true if you are discussing creating organisms which are highly different from existing ones. _delirium was discussing genetic modifications which are simply tweaks to existing organisms. There is no reason to believe that golden rice will evolve in any significantly different manner than ordinary rice. In contrast, AI will evolve via a mechanism which is unprecedented.
    - possibilistic 15y ago
      I imagine that this has already happened back when countries had active biowarfare programs: what if someone clones a mix of potent neurotoxins into bacteria or fungi? Or mosquitos? What if we create bacteria that can digest anything, survive in environmental extrema, sporulate, and kill off competing strains? The grey goo scenario comes to mind. Chemical and biological state space is infinite. There is much room for good, but also for bad. Misuse of biology is much more dangerous in the short term than AI.
  - jessriedel 15y ago
    I think that most people who take the AI singularity very seriously would say that genetics researchers should be held to similarly stringent standards, as should the handful of other research fields with significant existential risk (e.g. nanotech, high-energy physics). Eliezer (the interviewee) has an HN account so he can comment for himself.
- jordan0day 15y ago
  gnosis makes a good point, and to be honest, I'm not sure the AIs coming out of the AI research sector are the ones we'll need to be worried about anyway. Basically, I'm thinking something along the lines of Hanlon's Razor: "Never attribute to malice that which is adequately explained by stupidity." We'll see serious damage caused by "weak" AIs long before we have a "strong" AI capable of causing similar damage. For example, 2010's "Flash Crash" seems to have high frequency trading at its core. My hope is that through the growing pains we experience from "weak" AI systems doing something stupid, we'll be better prepared for a "strong" AI system that may try to do something malicious.
nazgulnarsil 15y ago
no. and the likelihood of someone building a friendly AI first when that is harder than building an unfriendly AI seems minuscule. cya humanity, sucked while it lasted anyway.
- MikeCapone 15y ago
  If you believe that, the rational thing to do is to support Friendly-AI research. SIAI.org
  - deleted 15y ago
    [deleted]
  - keefe 15y ago
    ...wonders if this is an SIAI affiliated human that I know... I should say something clever to the fatalist OP to avoid downmod... something about hardware requirements for agi likely being high (imho) and that we have a few decades to work on the really hard problem and we haven't yet worked on hard problems for a few decades since maturing as an information processing species.
aothman 15y ago
As an AI researcher, I think obstacles like "not having your robot fall over all the damn time" are a little more immediate than robots having a nuanced understanding of ethics. I can understand why this stuff is fun to think about and debate, but it's just not relevant at all to where AI is going to be for the next 50 (or 100, or probably 200) years.
- cryptoz 15y ago
  I really think the stability of your robot is a completely separate issue. George W. Bush and Barack Obama both use flying robots with missiles to hunt down and kill people they don't like. Don't you think that perhaps, as these flying robots gain more and more autonomy, that discussions of ethics are actually important, and important now? 50 years is a long, long time in computer science. I'm surprised that you are so pessimistic about your research that you think ethics won't even be relevant in year 2205. Holy cow you must think AI is hard.
  - jordan0day 15y ago
    George W. Bush and Barack Obama both use flying robots with missiles to hunt down and kill people they don't like. This is a very good point. It's always good to be reminded that we're already living in the future. That said, I feel like aothman is discussing real artificial intelligence, that is, an entity capable of making a conscious decision that it wants to, in this case, fire the missiles. If I had to guess, if predator drones gain the ability to "decide" for themselves whether or not to fire their missiles, it will be built on a system of complex rules, and not because they're "intelligent". Potayto, Potahto? Maybe. I'm not an AI researcher and I don't even come close to understanding human intelligence, but I feel like even if it is just a complex system of rules, it's at a much deeper level than we'll be able to simulate soon.
    - cabalamat 15y ago
      > It's always good to be reminded that we're already living in the future. This is not a new phenomenon; the first use of autonomous killer robots was in 1943, in the form of acoustically guided torpedoes.
HockeyBiasDotCo 15y ago
No. Not close yet...
geuis 15y ago
So I'm not a person that actively writes AI software, but I am a knowledgable supporter. I'm all for the Singularity, rights for future non-human intelligences, etc, et. al. So I always take issues with these kinds of esoteric debates about how to engineer ethics into an intelligence that can learn and become conscious. Haven't any of these yahoos ever had kids or owned a pet dog? You don't "engineer ethics" into your son or daughter. You teach them through examples of good behavior, punish them when they misbehave, and reward them when they succeed. Over the course of a few years, given a good environment, the end result is a new young intelligence that knows how to behave well and get along with others. That intelligence often goes on to bootstrap itself up into adulthood and eventually goes on to create later iterations of itself. If it was raised well, then the new ones tend to get raised well too. We call them "grandkids". So lets assume in 10-20 years something descended from IBM's Blue Brain (simulating cat cortexes) leads to something that is analogous in intellectual range from a dog to an elephant. Most people will agree that dogs and elephants are pretty damn smart. Dogs are able to perceive human emotional states, understand some language, do work for people, and fit nicely into our social structure. Elephants aren't that close with people, but are highly intelligent, have active internal emotional states, and even grieve for their dead. In some societies, people and elephants have worked together for thousands of years. In both these cases, we have a long history of working with other intelligences of varying scales for thousands of years. In general, if you don't mistreat them they turn out to be socialized pretty well. Its only when you mistreat them that they learn to fear and hate you. The same is true for people. So as @aothman said in another comment in this thread, AI researchers are just trying to get their projects to not fall over. There's no thought of "engineering ethics". This problem is going to be solved one little bit at a time. Artificial neural architectures are going to more and more sophisticated over time. But there is a key difference between the underlying architecture and how you go about training these new minds. If you raise them well, then most of these angels-on-a-pin discussions are just that, meaningless.
- curiousepic 15y ago
  There is no reason to believe the first AGI will be anthropomorphic (or even... animorphic(?)). Blue Brain is not the only path to AGI, and "raising" or "training" may have no analogues. http://lesswrong.com/lw/rm/the_design_space_of_mindsingeneral/ http://lesswrong.com/lw/rm/the_design_space_of_mindsingenera...
protomyth 15y ago
I don't think we understand debugging well enough to start building "friendly A.I.", much less in robot form.
- curiousepic 15y ago
  A true AGI will solve robotics itself, whether it is already instantiated in a robot or not.
A1kmm 15y ago
The thing is, fully autonomous AIs will most likely be tested on a simulated world (maybe at a smaller scale) before they have any kind of real world influence. Real world resources are finite, and real world processes with real world materials take a certain finite amount of time. The singularity therefore ignores the realities of physics. It would even be possible to add artificial constraints on the total resource use and rate of resource use.
Hipchan 15y ago
Haha, it's impossible. Absolute power corrupts absolutely. The goal is to make AI more powerful than humans is it not? We're not going to be able to control it, no way no how.

42 points by pldpld 15y ago

Best suggestion of the article is that we scorn AI researchers who do not have a credible claim that their designs will maintain a basic agreed-on value system after a billion self-managed iterations and upgrades by the AI. This is a fascinating and broad-ranging criticism of AI, and it's interesting to me because the author is clearly considering 'what happens if we are successful?'. Definitely worth a read.

gnosis 15y ago
"Best suggestion of the article is that we scorn AI researchers who do not have a credible claim that their designs will maintain a basic agreed-on value system after a billion self-managed iterations and upgrades by the AI." Unfortunately, I don't think that's really going to happen. When people refuse to fund research in a promising area, others wind up taking up the slack. Witness America's prohibition on fetal stem-cell research. The US lost its lead in that field when researchers from other countries continued the work. The same will be much more true of AI research, as a working strong AI would probably be seen as a goose that lays golden eggs. Of course, it's also a Pandora's Box. So we should be careful. But I don't think scorning researchers in the field is going to be very effective.
_delirium 15y ago
That's a pretty high bar that would be pretty controversial if you applied it to other fields. For example, should we scorn genetics researchers who do not have a credible claim that their organisms will remain harmless after a billion generations of evolution and recombination? That's more or less what many of the anti-GMO arguments boil down to, that we ought to require genetically-modified organisms to be provably safe, both as they exist now, and in all possible future ways they could evolve and interact with other organisms. (And since that bar is very hard to reach, therefore, their arguments go, we should be careful about funding such research to begin with, and definitely shouldn't let any of its results out into the wild, e.g. into crops.) If anything, the argument there is stronger, because evolving biological organisms that can pose a threat to humans actually exist, whereas evolving machines that can pose a threat to humans are sci-fi, and likely to remain so for a very long time. Why regulate the latter one more stringently?
- yummyfajitas 15y ago
  The search space of biological organisms been well explored. It's unlikely that one will develop which is vastly more dangerous than the ones which already exist. The same cannot be said for robots/AI.
  - jessriedel 15y ago
    > The search space of biological organisms been well explored. Not at all true. The space of possible biological organisms is searched in a highly nonuniform manner by evolution, and the human search strategy is fundamentally different. It's overwhelmingly likely that there competitive human-constructible organisms which could never be produced by evolution in the past 4 billion years.
    - yummyfajitas 15y ago
      This is true if you are discussing creating organisms which are highly different from existing ones. _delirium was discussing genetic modifications which are simply tweaks to existing organisms. There is no reason to believe that golden rice will evolve in any significantly different manner than ordinary rice. In contrast, AI will evolve via a mechanism which is unprecedented.
  - possibilistic 15y ago
    I imagine that this has already happened back when countries had active biowarfare programs: what if someone clones a mix of potent neurotoxins into bacteria or fungi? Or mosquitos? What if we create bacteria that can digest anything, survive in environmental extrema, sporulate, and kill off competing strains? The grey goo scenario comes to mind. Chemical and biological state space is infinite. There is much room for good, but also for bad. Misuse of biology is much more dangerous in the short term than AI.
- jessriedel 15y ago
  I think that most people who take the AI singularity very seriously would say that genetics researchers should be held to similarly stringent standards, as should the handful of other research fields with significant existential risk (e.g. nanotech, high-energy physics). Eliezer (the interviewee) has an HN account so he can comment for himself.
jordan0day 15y ago
gnosis makes a good point, and to be honest, I'm not sure the AIs coming out of the AI research sector are the ones we'll need to be worried about anyway. Basically, I'm thinking something along the lines of Hanlon's Razor: "Never attribute to malice that which is adequately explained by stupidity." We'll see serious damage caused by "weak" AIs long before we have a "strong" AI capable of causing similar damage. For example, 2010's "Flash Crash" seems to have high frequency trading at its core. My hope is that through the growing pains we experience from "weak" AI systems doing something stupid, we'll be better prepared for a "strong" AI system that may try to do something malicious.

nazgulnarsil 15y ago

no. and the likelihood of someone building a friendly AI first when that is harder than building an unfriendly AI seems minuscule. cya humanity, sucked while it lasted anyway.

MikeCapone 15y ago
If you believe that, the rational thing to do is to support Friendly-AI research. SIAI.org
- deleted 15y ago
  [deleted]
- keefe 15y ago
  ...wonders if this is an SIAI affiliated human that I know... I should say something clever to the fatalist OP to avoid downmod... something about hardware requirements for agi likely being high (imho) and that we have a few decades to work on the really hard problem and we haven't yet worked on hard problems for a few decades since maturing as an information processing species.

aothman 15y ago

As an AI researcher, I think obstacles like "not having your robot fall over all the damn time" are a little more immediate than robots having a nuanced understanding of ethics. I can understand why this stuff is fun to think about and debate, but it's just not relevant at all to where AI is going to be for the next 50 (or 100, or probably 200) years.

cryptoz 15y ago
I really think the stability of your robot is a completely separate issue. George W. Bush and Barack Obama both use flying robots with missiles to hunt down and kill people they don't like. Don't you think that perhaps, as these flying robots gain more and more autonomy, that discussions of ethics are actually important, and important now? 50 years is a long, long time in computer science. I'm surprised that you are so pessimistic about your research that you think ethics won't even be relevant in year 2205. Holy cow you must think AI is hard.
- jordan0day 15y ago
  George W. Bush and Barack Obama both use flying robots with missiles to hunt down and kill people they don't like. This is a very good point. It's always good to be reminded that we're already living in the future. That said, I feel like aothman is discussing real artificial intelligence, that is, an entity capable of making a conscious decision that it wants to, in this case, fire the missiles. If I had to guess, if predator drones gain the ability to "decide" for themselves whether or not to fire their missiles, it will be built on a system of complex rules, and not because they're "intelligent". Potayto, Potahto? Maybe. I'm not an AI researcher and I don't even come close to understanding human intelligence, but I feel like even if it is just a complex system of rules, it's at a much deeper level than we'll be able to simulate soon.
  - cabalamat 15y ago
    > It's always good to be reminded that we're already living in the future. This is not a new phenomenon; the first use of autonomous killer robots was in 1943, in the form of acoustically guided torpedoes.

HockeyBiasDotCo 15y ago

No. Not close yet...

geuis 15y ago

So I'm not a person that actively writes AI software, but I am a knowledgable supporter. I'm all for the Singularity, rights for future non-human intelligences, etc, et. al. So I always take issues with these kinds of esoteric debates about how to engineer ethics into an intelligence that can learn and become conscious. Haven't any of these yahoos ever had kids or owned a pet dog? You don't "engineer ethics" into your son or daughter. You teach them through examples of good behavior, punish them when they misbehave, and reward them when they succeed. Over the course of a few years, given a good environment, the end result is a new young intelligence that knows how to behave well and get along with others. That intelligence often goes on to bootstrap itself up into adulthood and eventually goes on to create later iterations of itself. If it was raised well, then the new ones tend to get raised well too. We call them "grandkids". So lets assume in 10-20 years something descended from IBM's Blue Brain (simulating cat cortexes) leads to something that is analogous in intellectual range from a dog to an elephant. Most people will agree that dogs and elephants are pretty damn smart. Dogs are able to perceive human emotional states, understand some language, do work for people, and fit nicely into our social structure. Elephants aren't that close with people, but are highly intelligent, have active internal emotional states, and even grieve for their dead. In some societies, people and elephants have worked together for thousands of years. In both these cases, we have a long history of working with other intelligences of varying scales for thousands of years. In general, if you don't mistreat them they turn out to be socialized pretty well. Its only when you mistreat them that they learn to fear and hate you. The same is true for people. So as @aothman said in another comment in this thread, AI researchers are just trying to get their projects to not fall over. There's no thought of "engineering ethics". This problem is going to be solved one little bit at a time. Artificial neural architectures are going to more and more sophisticated over time. But there is a key difference between the underlying architecture and how you go about training these new minds. If you raise them well, then most of these angels-on-a-pin discussions are just that, meaningless.

curiousepic 15y ago
There is no reason to believe the first AGI will be anthropomorphic (or even... animorphic(?)). Blue Brain is not the only path to AGI, and "raising" or "training" may have no analogues. http://lesswrong.com/lw/rm/the_design_space_of_mindsingeneral/ http://lesswrong.com/lw/rm/the_design_space_of_mindsingenera...

protomyth 15y ago

I don't think we understand debugging well enough to start building "friendly A.I.", much less in robot form.

curiousepic 15y ago
A true AGI will solve robotics itself, whether it is already instantiated in a robot or not.

A1kmm 15y ago

The thing is, fully autonomous AIs will most likely be tested on a simulated world (maybe at a smaller scale) before they have any kind of real world influence. Real world resources are finite, and real world processes with real world materials take a certain finite amount of time. The singularity therefore ignores the realities of physics. It would even be possible to add artificial constraints on the total resource use and rate of resource use.

Hipchan 15y ago

Haha, it's impossible. Absolute power corrupts absolutely. The goal is to make AI more powerful than humans is it not? We're not going to be able to control it, no way no how.