3 ms·

Best suggestion of the article is that we scorn AI researchers who do not have a credible claim that their designs will maintain a basic agreed-on value system

Best suggestion of the article is that we scorn AI researchers who do not have a credible claim that their designs will maintain a basic agreed-on value system after a billion self-managed iterations and upgrades by the AI. This is a fascinating and broad-ranging criticism of AI, and it's interesting to me because the author is clearly considering 'what happens if we are successful?'. Definitely worth a read.

gnosis 16y ago
"Best suggestion of the article is that we scorn AI researchers who do not have a credible claim that their designs will maintain a basic agreed-on value system after a billion self-managed iterations and upgrades by the AI." Unfortunately, I don't think that's really going to happen. When people refuse to fund research in a promising area, others wind up taking up the slack. Witness America's prohibition on fetal stem-cell research. The US lost its lead in that field when researchers from other countries continued the work. The same will be much more true of AI research, as a working strong AI would probably be seen as a goose that lays golden eggs. Of course, it's also a Pandora's Box. So we should be careful. But I don't think scorning researchers in the field is going to be very effective.
_delirium 16y ago
That's a pretty high bar that would be pretty controversial if you applied it to other fields. For example, should we scorn genetics researchers who do not have a credible claim that their organisms will remain harmless after a billion generations of evolution and recombination? That's more or less what many of the anti-GMO arguments boil down to, that we ought to require genetically-modified organisms to be provably safe, both as they exist now, and in all possible future ways they could evolve and interact with other organisms. (And since that bar is very hard to reach, therefore, their arguments go, we should be careful about funding such research to begin with, and definitely shouldn't let any of its results out into the wild, e.g. into crops.) If anything, the argument there is stronger, because evolving biological organisms that can pose a threat to humans actually exist, whereas evolving machines that can pose a threat to humans are sci-fi, and likely to remain so for a very long time. Why regulate the latter one more stringently?
- yummyfajitas 16y ago
  The search space of biological organisms been well explored. It's unlikely that one will develop which is vastly more dangerous than the ones which already exist. The same cannot be said for robots/AI.
  - jessriedel 16y ago
    > The search space of biological organisms been well explored. Not at all true. The space of possible biological organisms is searched in a highly nonuniform manner by evolution, and the human search strategy is fundamentally different. It's overwhelmingly likely that there competitive human-constructible organisms which could never be produced by evolution in the past 4 billion years.
    - yummyfajitas 16y ago
      This is true if you are discussing creating organisms which are highly different from existing ones. _delirium was discussing genetic modifications which are simply tweaks to existing organisms. There is no reason to believe that golden rice will evolve in any significantly different manner than ordinary rice. In contrast, AI will evolve via a mechanism which is unprecedented.
  - possibilistic 16y ago
    I imagine that this has already happened back when countries had active biowarfare programs: what if someone clones a mix of potent neurotoxins into bacteria or fungi? Or mosquitos? What if we create bacteria that can digest anything, survive in environmental extrema, sporulate, and kill off competing strains? The grey goo scenario comes to mind. Chemical and biological state space is infinite. There is much room for good, but also for bad. Misuse of biology is much more dangerous in the short term than AI.
- jessriedel 16y ago
  I think that most people who take the AI singularity very seriously would say that genetics researchers should be held to similarly stringent standards, as should the handful of other research fields with significant existential risk (e.g. nanotech, high-energy physics). Eliezer (the interviewee) has an HN account so he can comment for himself.
jordan0day 16y ago
gnosis makes a good point, and to be honest, I'm not sure the AIs coming out of the AI research sector are the ones we'll need to be worried about anyway. Basically, I'm thinking something along the lines of Hanlon's Razor: "Never attribute to malice that which is adequately explained by stupidity." We'll see serious damage caused by "weak" AIs long before we have a "strong" AI capable of causing similar damage. For example, 2010's "Flash Crash" seems to have high frequency trading at its core. My hope is that through the growing pains we experience from "weak" AI systems doing something stupid, we'll be better prepared for a "strong" AI system that may try to do something malicious.

gnosis 16y ago

"Best suggestion of the article is that we scorn AI researchers who do not have a credible claim that their designs will maintain a basic agreed-on value system after a billion self-managed iterations and upgrades by the AI." Unfortunately, I don't think that's really going to happen. When people refuse to fund research in a promising area, others wind up taking up the slack. Witness America's prohibition on fetal stem-cell research. The US lost its lead in that field when researchers from other countries continued the work. The same will be much more true of AI research, as a working strong AI would probably be seen as a goose that lays golden eggs. Of course, it's also a Pandora's Box. So we should be careful. But I don't think scorning researchers in the field is going to be very effective.

_delirium 16y ago

That's a pretty high bar that would be pretty controversial if you applied it to other fields. For example, should we scorn genetics researchers who do not have a credible claim that their organisms will remain harmless after a billion generations of evolution and recombination? That's more or less what many of the anti-GMO arguments boil down to, that we ought to require genetically-modified organisms to be provably safe, both as they exist now, and in all possible future ways they could evolve and interact with other organisms. (And since that bar is very hard to reach, therefore, their arguments go, we should be careful about funding such research to begin with, and definitely shouldn't let any of its results out into the wild, e.g. into crops.) If anything, the argument there is stronger, because evolving biological organisms that can pose a threat to humans actually exist, whereas evolving machines that can pose a threat to humans are sci-fi, and likely to remain so for a very long time. Why regulate the latter one more stringently?

yummyfajitas 16y ago
The search space of biological organisms been well explored. It's unlikely that one will develop which is vastly more dangerous than the ones which already exist. The same cannot be said for robots/AI.
- jessriedel 16y ago
  > The search space of biological organisms been well explored. Not at all true. The space of possible biological organisms is searched in a highly nonuniform manner by evolution, and the human search strategy is fundamentally different. It's overwhelmingly likely that there competitive human-constructible organisms which could never be produced by evolution in the past 4 billion years.
  - yummyfajitas 16y ago
    This is true if you are discussing creating organisms which are highly different from existing ones. _delirium was discussing genetic modifications which are simply tweaks to existing organisms. There is no reason to believe that golden rice will evolve in any significantly different manner than ordinary rice. In contrast, AI will evolve via a mechanism which is unprecedented.
- possibilistic 16y ago
  I imagine that this has already happened back when countries had active biowarfare programs: what if someone clones a mix of potent neurotoxins into bacteria or fungi? Or mosquitos? What if we create bacteria that can digest anything, survive in environmental extrema, sporulate, and kill off competing strains? The grey goo scenario comes to mind. Chemical and biological state space is infinite. There is much room for good, but also for bad. Misuse of biology is much more dangerous in the short term than AI.
jessriedel 16y ago
I think that most people who take the AI singularity very seriously would say that genetics researchers should be held to similarly stringent standards, as should the handful of other research fields with significant existential risk (e.g. nanotech, high-energy physics). Eliezer (the interviewee) has an HN account so he can comment for himself.

jordan0day 16y ago

gnosis makes a good point, and to be honest, I'm not sure the AIs coming out of the AI research sector are the ones we'll need to be worried about anyway. Basically, I'm thinking something along the lines of Hanlon's Razor: "Never attribute to malice that which is adequately explained by stupidity." We'll see serious damage caused by "weak" AIs long before we have a "strong" AI capable of causing similar damage. For example, 2010's "Flash Crash" seems to have high frequency trading at its core. My hope is that through the growing pains we experience from "weak" AI systems doing something stupid, we'll be better prepared for a "strong" AI system that may try to do something malicious.