2 ms·
AI risk has a profile that is very, very difficult to deal with rationally. By that I don't just mean "think about rationally rather than emotionally", but that
by jerf 8y ago
AI risk has a profile that is very, very difficult to deal with rationally. By that I don't just mean "think about rationally rather than emotionally", but that it is difficult to even use rational tools to analyze it. It is what most people would consider a very, very small probability of what is arguably the worst possible outcome (i.e., considered from a strictly materialist perspectiv, there are outcomes worse that "the total extinction of life on Earth"!). Tiny magnitude probabilities of huge magnitude disasters are mathematically unstable; very small shifts in the value of the probability, and relatively small changes in the log value of the disaster's size, result in radically different scenarios and correct risk mitigations.
It's even worse if you try to draw the probability distribution of something like "given strong AI will be acheived within the next 100 years, what is the distribution of the resulting likely outcomes?" In this case, the uncertainties are so large that, again, people can come to radically different conclusions even assuming both sides are being generally rational. It asks people trying to draw that distribution to consider what the most likely outcome is of at least a radically super-intelligent being is, and more likely, an arbitrarily large community of radically super-intelligent beings. Who can seriously claim to have an accurate probability distribution of that? If we understood radically super-intelligent beings, we'd already be radically super-intelligent beings, so we are not good at modeling them almost by definition.
But certainly by observation; I am opaque in many ways even to my 7 and 10 year old children, and as with most humans, they are already exceptionally intelligent as "intelligent systems" go. (A 5 year old of normal intelligence is already exceptionally intelligent as intelligent systems go.) The idea that I could model something that was even my clone otherwise but operating at a hundred times the speed is already absurd; add an actual increase in intelligence to it and I stand no chance. Our only commonality will be those things imposed on us by the universe; i.e., I can be confident it must consume some negentropy to survive, etc. But any higher-level actions would be impossible for me to model.
People who think intelligence is unlike to emerge quickly are unlikely to consider it a serious risk. People who think it's unlikely for an intelligent being to augment itself, then use its augmentations to augment itself, in an explosion of uncontrollable intelligence, will not consider AI risk a very likely outcome. And it's not as if it's an irrational or impossible outcome; for all we know, while there may be low-hanging fruit above human intelligence it is absolutely entirely possible that O(work to increase intelligence) > O(ability to perform intelligence-increasing work as intelligence goes up). After all, I can't help but look up in the sky and observe that there does not seem to be a near-lightspeed expanding bubble of computronium coming my way, per the most dramatic fears of the singularity. We could end up with something very intelligent, even dangerously intelligent, but not end up with us waking up one day to a digital god among us.
- justinpombrio 8y agoFYI, the majority of your arguments are explicitly discussed in Scott Alexander's post.
- Veedrac 8y agoThe precise distribution of outcomes is not actually all that important for just figuring out whether the research is valuable. Alignment research would be valuable if it moved a 1% probability mass from worst-possible-world to human-extinction, a 1% probability mass from human-extinction to human-survival, or a 1% probability mass from human-survival to human-flourishing. As long as you can convince yourself that there is a probability of things being far from optimal in some sense at some time in a way that funding research now could nontrivially affect, you have a justification. Remember that people were created by evolution, and evolution is dumb. Nonetheless, the returns on moderate deltas in intelligence are very strongly selected for, and major changes have happened with small perturbations in evolutionary history. This gradient is so sharp that the same species that produces perfectly healthy 80 IQ individuals also produces Feynman and Euler, the latter so mathematically productive that the Wikipedia page listing things named after him says that "In an effort to avoid naming everything after Euler, some discoveries and theorems are attributed to the first person to have proved them after Euler." Note that necessarily society develops as soon as intelligence reaches the point at which is is achievable, not with some evolutionarily significant delay, so you cannot judge the gradient by looking at the cap on intelligence of existing species; we are the necessarily the smartest, because we are first. Rather, you have to look at those behind us, and there it seems the gradient is extremely steep; our closest intellectual competitors can not so much as write a single coherent sentence. The probable conclusion is that human intelligence is more than sufficient to build effective general intelligence, and improvements to intelligence are likely extremely steep on the slope to superintelligence, even ignoring the nine orders of magnitude improvement you get for free by merely running in silicon.