3 ms·

There are blind alleys, and then there are ideas that make people want to psychoanalyze you. To hyperbolize a bit, but only a bit, worrying about runaway AI tha

by arakyd 17y ago

There are blind alleys, and then there are ideas that make people want to psychoanalyze you. To hyperbolize a bit, but only a bit, worrying about runaway AI that has the potential to turn the universe into paperclips is about as useful as worrying about warp drives that damage spacetime. It's hard enough to create fictional technologies that require basic changes in our understanding of the laws of physics, let alone debug them before they are created, let alone debug them before we make the basic discoveries that make them possible, let alone make them provably safe before we make the basic discoveries that make them possible and before we even figure out how to prove that relatively simple programs won't crash. I know of only one science fiction author who was honest enough to spell out the fact that his hypothetical runaway AI had to make several foundational breakthroughs in physics at just the right time in order to plausibly take off faster than anyone could stop it, or become as smart and powerful as singularitarians assume is possible. We have no evidence that the laws of physics in our universe allow that sort of thing. Most super-AI fiction (and everything written about super-AI is fiction at this point) either glosses over this by tacitly assuming that the plodding nature of human intelligence is the only real bottleneck to godhood (most transhumanists), or assumes that large numbers of humans will be incredibly incompetent (most mainstream AI fiction).

stcredzero 17y ago
before we even figure out how to prove that relatively simple programs won't crash You are forgetting the lessons of pragmatism. If AI can get a heuristic sense of whether a simple program will or won't crash with 99.999% accuracy in practice, then you already have a very useful tool. What if the AI we produce turns out to to be a bit dimmer than us, but never tires, can reproduce just by making copies, and is tirelessly diligent? I suspect that such an AI could beat us in a war, despite our being arguably "smarter."
- sho 17y ago
  What if? Well then we'd pull out its power plug. Or build another AI and program it to be sympathetic to our needs. Or any of an infinite number of possible solutions, assuming we were the utter idiots necessary to not implement appropriate safeguards in the first place. Look, anyone can come up with an infinite number of movie-plot scenarios where naive humans bumbled into technology that then destroyed them. You can say that about almost anything. In understanding DNA, we might accidentally create an unstoppable virus! In conducting space exploration, we may alert a hostile alien civilisation to our presence! In researching chemistry we may create ice-9! Etc etc etc until the end of time. All of these, including EY's thesis, have one thing in common - the unjustified, massive expansion of a speculative and highly unlikely risk into a reason for retarding progress which would otherwise yield non-speculative, highly likely, extraordinarily beneficial gains.
  - stcredzero 17y ago
    What if? Well then we'd pull out its power plug. I was just posing an interesting question, especially as regards to the whole notion of "superior" intelligence. Superior in what regards? By whose measure? Eliezer should be given credit for promoting an information theoretic approach to that question. At least that seems to be a fundamental measure with a good chance of escaping cultural biases concerning "intelligence." I am certainly not in some simpleminded superior AI is going to kill us all camp. Perhaps we will have very powerful AI optimization tools that have no sense of self, or self-originating volition whatsoever. There would be no reason for such entities to act in their own self interest, and therefore no danger for their interests to conflict with our own. They would have the disadvantage, though, of never coming up with something neat on their own initiative. I think there's enough more than enough initiative from human sources. What's needed is better optimization. (Of course, only one rogue self-directed AI entity escaping into the wild could possibly -- not certainly -- doom us all. But this is not a new kind of danger. We have been facing that sort of danger -- where one robust and virulent enough example could escape and wreak havoc -- from technologies based on molecular biology for a few years now. So far, so good.)
- arakyd 17y ago
  FAI advocates are not satisfied with a 99.999% chance that the universe will not be turned into a uniform distribution of paperclips.
  - Eliezer 17y ago
    99.999%? I'd totally take it.- if it was a real number, calculated by some trustworthy method, and not a bogus expert estimate. Of course if you can do that, probability ~1 (i.e., proven theorem given that transistors obey stated axioms) is probably just as easy.

by arakyd 17y ago

stcredzero 17y ago

before we even figure out how to prove that relatively simple programs won't crash You are forgetting the lessons of pragmatism. If AI can get a heuristic sense of whether a simple program will or won't crash with 99.999% accuracy in practice, then you already have a very useful tool. What if the AI we produce turns out to to be a bit dimmer than us, but never tires, can reproduce just by making copies, and is tirelessly diligent? I suspect that such an AI could beat us in a war, despite our being arguably "smarter."

sho 17y ago
What if? Well then we'd pull out its power plug. Or build another AI and program it to be sympathetic to our needs. Or any of an infinite number of possible solutions, assuming we were the utter idiots necessary to not implement appropriate safeguards in the first place. Look, anyone can come up with an infinite number of movie-plot scenarios where naive humans bumbled into technology that then destroyed them. You can say that about almost anything. In understanding DNA, we might accidentally create an unstoppable virus! In conducting space exploration, we may alert a hostile alien civilisation to our presence! In researching chemistry we may create ice-9! Etc etc etc until the end of time. All of these, including EY's thesis, have one thing in common - the unjustified, massive expansion of a speculative and highly unlikely risk into a reason for retarding progress which would otherwise yield non-speculative, highly likely, extraordinarily beneficial gains.
- stcredzero 17y ago
  What if? Well then we'd pull out its power plug. I was just posing an interesting question, especially as regards to the whole notion of "superior" intelligence. Superior in what regards? By whose measure? Eliezer should be given credit for promoting an information theoretic approach to that question. At least that seems to be a fundamental measure with a good chance of escaping cultural biases concerning "intelligence." I am certainly not in some simpleminded superior AI is going to kill us all camp. Perhaps we will have very powerful AI optimization tools that have no sense of self, or self-originating volition whatsoever. There would be no reason for such entities to act in their own self interest, and therefore no danger for their interests to conflict with our own. They would have the disadvantage, though, of never coming up with something neat on their own initiative. I think there's enough more than enough initiative from human sources. What's needed is better optimization. (Of course, only one rogue self-directed AI entity escaping into the wild could possibly -- not certainly -- doom us all. But this is not a new kind of danger. We have been facing that sort of danger -- where one robust and virulent enough example could escape and wreak havoc -- from technologies based on molecular biology for a few years now. So far, so good.)
arakyd 17y ago
FAI advocates are not satisfied with a 99.999% chance that the universe will not be turned into a uniform distribution of paperclips.
- Eliezer 17y ago
  99.999%? I'd totally take it.- if it was a real number, calculated by some trustworthy method, and not a bogus expert estimate. Of course if you can do that, probability ~1 (i.e., proven theorem given that transistors obey stated axioms) is probably just as easy.