5 ms·

Is there really “a lot of theory” that says runaway AI is possible? In the sense of empirical fact-based peer reviewed machine learning literature? Because if

Is there really “a lot of theory” that says runaway AI is possible? In the sense of empirical fact-based peer reviewed machine learning literature? Because if so I must have missed it. It seems more accurate to say there is quite a bit of writing done by vocal influencers who frequent a couple online forums.

pixl97 3y ago
In evolutionary biological systems 'capabilities break out' common enough in viral and bacterial systems with their fast evolution times. The same thing can occur when new predators are introduced to ecosystems that are not prepared for them. The question that I assume you have is 'can AI systems experience large evolutionary jumps in capabilities'. I would say yes myself, but you apper to disagree.
DirkH 3y ago
There was also only "a lot of theory" and zero empirical evidence that nuclear bombs were possible not that long ago. Theoretical fact-based peer reviewed literature often (though not always) precedes empirical evidence. We shouldn't jump the gun and say runaway AI is inevitable just because there is fact-based theory on its possibility. Just like it would have been dumb to just look at E=mc^2 and conclude nuclear weapon development is 100% inevitable. But in the same breath we shouldn't say it's impossible just because we don't have empirical evidence either.
- throwaway9274 3y ago
  Prior to the development of nuclear weapons, there was well-grounded peer reviewed theory in the physics literature. There were predictions that could be made by the theoretical frameworks, and experiments to confirm those predictions. The literature well described the composition of atoms, the decay of radioactive elements, the concept of fission, and the conversion of mass to energy. No such groundwork exists to substantiate the claim that runaway AI is possible. LessWrong, the so-called Alignment Forum, and similar Internet groups propose thought experiments. They couch these thought experiments in the guise of inevitable outcomes. The prophesies of the influencers originating there have no more relationship to the truth than the average enthusiast’s daydreams. The content there fills a role similar to science fiction in the atomic age: fanciful speculation on the properties of imaginary systems. Over time, the bad content is forgotten, and a handful of items survive and look prescient via selection bias.
  - DirkH 3y ago
    Don't we likewise have predictions about FLOPs, compute, algorithms etc etc that are grounded in theoretical frameworks, and every year or so we make predictions and make experiments to confirm or deny AI capability predictions? The AI and neuroscience literature likewise describe the composition of intelligence and what it takes to build it. What would you need to see to believe we have enough evidence that runaway AI is possible, even if it is highly unlikely any time soon? (Worth adding here I don't know exactly what you mean by runaway AI other than simply AI humanity as a whole loses control over). I'm asking what it would take for you to go from 0% possible to 5% possible. Last time I checked science fiction authors in the atomic age making fanciful speculation on the properties of imaginary systems weren't responsible for actually building said systems. Though naturally, there will be exceptions. Nonetheless, I still have a hard time dismissing Geoffrey Hinton's concerns as fanciful sci-fi. Likewise, a nuclear world war actually wouldn't be as bad or even civilization-ending as fanciful sci-fi authors make it out to be. But dismissing its risks entirely because there are some nuclear-war forums where people come up with weird radiation thought experiments just seems preemptive to me.

pixl97 3y ago

In evolutionary biological systems 'capabilities break out' common enough in viral and bacterial systems with their fast evolution times. The same thing can occur when new predators are introduced to ecosystems that are not prepared for them. The question that I assume you have is 'can AI systems experience large evolutionary jumps in capabilities'. I would say yes myself, but you apper to disagree.

DirkH 3y ago

There was also only "a lot of theory" and zero empirical evidence that nuclear bombs were possible not that long ago. Theoretical fact-based peer reviewed literature often (though not always) precedes empirical evidence. We shouldn't jump the gun and say runaway AI is inevitable just because there is fact-based theory on its possibility. Just like it would have been dumb to just look at E=mc^2 and conclude nuclear weapon development is 100% inevitable. But in the same breath we shouldn't say it's impossible just because we don't have empirical evidence either.

throwaway9274 3y ago
Prior to the development of nuclear weapons, there was well-grounded peer reviewed theory in the physics literature. There were predictions that could be made by the theoretical frameworks, and experiments to confirm those predictions. The literature well described the composition of atoms, the decay of radioactive elements, the concept of fission, and the conversion of mass to energy. No such groundwork exists to substantiate the claim that runaway AI is possible. LessWrong, the so-called Alignment Forum, and similar Internet groups propose thought experiments. They couch these thought experiments in the guise of inevitable outcomes. The prophesies of the influencers originating there have no more relationship to the truth than the average enthusiast’s daydreams. The content there fills a role similar to science fiction in the atomic age: fanciful speculation on the properties of imaginary systems. Over time, the bad content is forgotten, and a handful of items survive and look prescient via selection bias.
- DirkH 3y ago
  Don't we likewise have predictions about FLOPs, compute, algorithms etc etc that are grounded in theoretical frameworks, and every year or so we make predictions and make experiments to confirm or deny AI capability predictions? The AI and neuroscience literature likewise describe the composition of intelligence and what it takes to build it. What would you need to see to believe we have enough evidence that runaway AI is possible, even if it is highly unlikely any time soon? (Worth adding here I don't know exactly what you mean by runaway AI other than simply AI humanity as a whole loses control over). I'm asking what it would take for you to go from 0% possible to 5% possible. Last time I checked science fiction authors in the atomic age making fanciful speculation on the properties of imaginary systems weren't responsible for actually building said systems. Though naturally, there will be exceptions. Nonetheless, I still have a hard time dismissing Geoffrey Hinton's concerns as fanciful sci-fi. Likewise, a nuclear world war actually wouldn't be as bad or even civilization-ending as fanciful sci-fi authors make it out to be. But dismissing its risks entirely because there are some nuclear-war forums where people come up with weird radiation thought experiments just seems preemptive to me.