7 ms·

> However, to consider artificial intelligence to be a potential global catastrophe at all, let alone the single one requiring extra funding, is mostly unfounde

> However, to consider artificial intelligence to be a potential global catastrophe at all, let alone the single one requiring extra funding, is mostly unfounded. As another commenter points out, EA is not a single unified group, but rather a disparate set of people who are united by the belief that we should put money where it has the largest effect. Some, but not all, of them are convinced by the arguments in favor of doing AI alignment research. If you aren't, then don't donate money for that, donate it to what you think is a more effective cause! I'm curious, though, how people come to the conclusion that AI risk is nonsense? Is it a gut reaction, or does it come from thinking or reading about the problem? Every popsci article I've seen on the topic has been atrocious, so kudos if you thought "nonsense!" after reading such an article. Scott Alexander has a pretty readable introduction to AI risk: http://slatestarcodex.com/superintelligence-faq/ http://slatestarcodex.com/superintelligence-faq/

jerf 8y ago
AI risk has a profile that is very, very difficult to deal with rationally. By that I don't just mean "think about rationally rather than emotionally", but that it is difficult to even use rational tools to analyze it. It is what most people would consider a very, very small probability of what is arguably the worst possible outcome (i.e., considered from a strictly materialist perspectiv, there are outcomes worse that "the total extinction of life on Earth"!). Tiny magnitude probabilities of huge magnitude disasters are mathematically unstable; very small shifts in the value of the probability, and relatively small changes in the log value of the disaster's size, result in radically different scenarios and correct risk mitigations. It's even worse if you try to draw the probability distribution of something like "given strong AI will be acheived within the next 100 years, what is the distribution of the resulting likely outcomes?" In this case, the uncertainties are so large that, again, people can come to radically different conclusions even assuming both sides are being generally rational. It asks people trying to draw that distribution to consider what the most likely outcome is of at least a radically super-intelligent being is, and more likely, an arbitrarily large community of radically super-intelligent beings. Who can seriously claim to have an accurate probability distribution of that? If we understood radically super-intelligent beings, we'd already be radically super-intelligent beings, so we are not good at modeling them almost by definition. But certainly by observation; I am opaque in many ways even to my 7 and 10 year old children, and as with most humans, they are already exceptionally intelligent as "intelligent systems" go. (A 5 year old of normal intelligence is already exceptionally intelligent as intelligent systems go.) The idea that I could model something that was even my clone otherwise but operating at a hundred times the speed is already absurd; add an actual increase in intelligence to it and I stand no chance. Our only commonality will be those things imposed on us by the universe; i.e., I can be confident it must consume some negentropy to survive, etc. But any higher-level actions would be impossible for me to model. People who think intelligence is unlike to emerge quickly are unlikely to consider it a serious risk. People who think it's unlikely for an intelligent being to augment itself, then use its augmentations to augment itself, in an explosion of uncontrollable intelligence, will not consider AI risk a very likely outcome. And it's not as if it's an irrational or impossible outcome; for all we know, while there may be low-hanging fruit above human intelligence it is absolutely entirely possible that O(work to increase intelligence) > O(ability to perform intelligence-increasing work as intelligence goes up). After all, I can't help but look up in the sky and observe that there does not seem to be a near-lightspeed expanding bubble of computronium coming my way, per the most dramatic fears of the singularity. We could end up with something very intelligent, even dangerously intelligent, but not end up with us waking up one day to a digital god among us.
- justinpombrio 8y ago
  FYI, the majority of your arguments are explicitly discussed in Scott Alexander's post.
- Veedrac 8y ago
  The precise distribution of outcomes is not actually all that important for just figuring out whether the research is valuable. Alignment research would be valuable if it moved a 1% probability mass from worst-possible-world to human-extinction, a 1% probability mass from human-extinction to human-survival, or a 1% probability mass from human-survival to human-flourishing. As long as you can convince yourself that there is a probability of things being far from optimal in some sense at some time in a way that funding research now could nontrivially affect, you have a justification. Remember that people were created by evolution, and evolution is dumb. Nonetheless, the returns on moderate deltas in intelligence are very strongly selected for, and major changes have happened with small perturbations in evolutionary history. This gradient is so sharp that the same species that produces perfectly healthy 80 IQ individuals also produces Feynman and Euler, the latter so mathematically productive that the Wikipedia page listing things named after him says that "In an effort to avoid naming everything after Euler, some discoveries and theorems are attributed to the first person to have proved them after Euler." Note that necessarily society develops as soon as intelligence reaches the point at which is is achievable, not with some evolutionarily significant delay, so you cannot judge the gradient by looking at the cap on intelligence of existing species; we are the necessarily the smartest, because we are first. Rather, you have to look at those behind us, and there it seems the gradient is extremely steep; our closest intellectual competitors can not so much as write a single coherent sentence. The probable conclusion is that human intelligence is more than sufficient to build effective general intelligence, and improvements to intelligence are likely extremely steep on the slope to superintelligence, even ignoring the nine orders of magnitude improvement you get for free by merely running in silicon.
probably_wrong 8y ago
I assume that investing in AI risk is nonsense for one simple reason: I have yet to see a single study showing that a single undesired behavior has been stopped or even slowed down. DDOS attacks are as stupid as you can get from an intelligence point of view, and yet not a single proponent for AI risk has come up with a way to stop even a single one AFAIK. I haven't heard either of a single military program that has been thwarted, and those computers are killing people right now. I also doubt that they could get a Pentagon official to stop doing anything. Until the AI risk community show that they can do anything other than talk, no matter how small, I'll remain skeptic.
- deleted 8y ago
  [deleted]
- justinpombrio 8y ago
  I'm confused. The research in question is "how can we make sure that a superintelligence would be morally aligned with humans". Why would dealing with DDOS attacks would be the right first step? It seems completely unrelated.
  - probably_wrong 8y ago
    Because (IMHO) if they want to morally align a superintelligence, the first step would be to either morally align a dumb intelligence or to show that you can convince the people building this intelligence to steer it in this direction. If they do neither, they risk either coming up with a plan that doesn't work (because it was not tested) or that no one cares about.
currymj 8y ago
i don't think it's nonsense exactly. i think it merits some funding, and we'll probably be glad we did it even if it turns out there never was a risk of an unfriendly superintelligence. but I have misgivings. in particular, the idea that marginal charitable dollars ought to be used to fund philosophers to think abstract thoughts is very counterintuitive! (every argument that it's THE MOST IMPORTANT THING IN THE WORLD seems to have the same form. first give me a bunch of weird historical and metaphysical assumptions, then make me admit I'd assign them a finite probability of being true, then multiply by a kajillion future simulated lives or whatever.) most of the philosophers who research this stuff, and most of the people involved in Effective Altruism, are part of the same niche subculture. i don't think this is a grift, I think they are quite honest in their convictions. but it makes me go "hmmmmm".

jerf 8y ago

AI risk has a profile that is very, very difficult to deal with rationally. By that I don't just mean "think about rationally rather than emotionally", but that it is difficult to even use rational tools to analyze it. It is what most people would consider a very, very small probability of what is arguably the worst possible outcome (i.e., considered from a strictly materialist perspectiv, there are outcomes worse that "the total extinction of life on Earth"!). Tiny magnitude probabilities of huge magnitude disasters are mathematically unstable; very small shifts in the value of the probability, and relatively small changes in the log value of the disaster's size, result in radically different scenarios and correct risk mitigations. It's even worse if you try to draw the probability distribution of something like "given strong AI will be acheived within the next 100 years, what is the distribution of the resulting likely outcomes?" In this case, the uncertainties are so large that, again, people can come to radically different conclusions even assuming both sides are being generally rational. It asks people trying to draw that distribution to consider what the most likely outcome is of at least a radically super-intelligent being is, and more likely, an arbitrarily large community of radically super-intelligent beings. Who can seriously claim to have an accurate probability distribution of that? If we understood radically super-intelligent beings, we'd already be radically super-intelligent beings, so we are not good at modeling them almost by definition. But certainly by observation; I am opaque in many ways even to my 7 and 10 year old children, and as with most humans, they are already exceptionally intelligent as "intelligent systems" go. (A 5 year old of normal intelligence is already exceptionally intelligent as intelligent systems go.) The idea that I could model something that was even my clone otherwise but operating at a hundred times the speed is already absurd; add an actual increase in intelligence to it and I stand no chance. Our only commonality will be those things imposed on us by the universe; i.e., I can be confident it must consume some negentropy to survive, etc. But any higher-level actions would be impossible for me to model. People who think intelligence is unlike to emerge quickly are unlikely to consider it a serious risk. People who think it's unlikely for an intelligent being to augment itself, then use its augmentations to augment itself, in an explosion of uncontrollable intelligence, will not consider AI risk a very likely outcome. And it's not as if it's an irrational or impossible outcome; for all we know, while there may be low-hanging fruit above human intelligence it is absolutely entirely possible that O(work to increase intelligence) > O(ability to perform intelligence-increasing work as intelligence goes up). After all, I can't help but look up in the sky and observe that there does not seem to be a near-lightspeed expanding bubble of computronium coming my way, per the most dramatic fears of the singularity. We could end up with something very intelligent, even dangerously intelligent, but not end up with us waking up one day to a digital god among us.

justinpombrio 8y ago
FYI, the majority of your arguments are explicitly discussed in Scott Alexander's post.
Veedrac 8y ago
The precise distribution of outcomes is not actually all that important for just figuring out whether the research is valuable. Alignment research would be valuable if it moved a 1% probability mass from worst-possible-world to human-extinction, a 1% probability mass from human-extinction to human-survival, or a 1% probability mass from human-survival to human-flourishing. As long as you can convince yourself that there is a probability of things being far from optimal in some sense at some time in a way that funding research now could nontrivially affect, you have a justification. Remember that people were created by evolution, and evolution is dumb. Nonetheless, the returns on moderate deltas in intelligence are very strongly selected for, and major changes have happened with small perturbations in evolutionary history. This gradient is so sharp that the same species that produces perfectly healthy 80 IQ individuals also produces Feynman and Euler, the latter so mathematically productive that the Wikipedia page listing things named after him says that "In an effort to avoid naming everything after Euler, some discoveries and theorems are attributed to the first person to have proved them after Euler." Note that necessarily society develops as soon as intelligence reaches the point at which is is achievable, not with some evolutionarily significant delay, so you cannot judge the gradient by looking at the cap on intelligence of existing species; we are the necessarily the smartest, because we are first. Rather, you have to look at those behind us, and there it seems the gradient is extremely steep; our closest intellectual competitors can not so much as write a single coherent sentence. The probable conclusion is that human intelligence is more than sufficient to build effective general intelligence, and improvements to intelligence are likely extremely steep on the slope to superintelligence, even ignoring the nine orders of magnitude improvement you get for free by merely running in silicon.

probably_wrong 8y ago

I assume that investing in AI risk is nonsense for one simple reason: I have yet to see a single study showing that a single undesired behavior has been stopped or even slowed down. DDOS attacks are as stupid as you can get from an intelligence point of view, and yet not a single proponent for AI risk has come up with a way to stop even a single one AFAIK. I haven't heard either of a single military program that has been thwarted, and those computers are killing people right now. I also doubt that they could get a Pentagon official to stop doing anything. Until the AI risk community show that they can do anything other than talk, no matter how small, I'll remain skeptic.

deleted 8y ago
[deleted]
justinpombrio 8y ago
I'm confused. The research in question is "how can we make sure that a superintelligence would be morally aligned with humans". Why would dealing with DDOS attacks would be the right first step? It seems completely unrelated.
- probably_wrong 8y ago
  Because (IMHO) if they want to morally align a superintelligence, the first step would be to either morally align a dumb intelligence or to show that you can convince the people building this intelligence to steer it in this direction. If they do neither, they risk either coming up with a plan that doesn't work (because it was not tested) or that no one cares about.

currymj 8y ago

i don't think it's nonsense exactly. i think it merits some funding, and we'll probably be glad we did it even if it turns out there never was a risk of an unfriendly superintelligence. but I have misgivings. in particular, the idea that marginal charitable dollars ought to be used to fund philosophers to think abstract thoughts is very counterintuitive! (every argument that it's THE MOST IMPORTANT THING IN THE WORLD seems to have the same form. first give me a bunch of weird historical and metaphysical assumptions, then make me admit I'd assign them a finite probability of being true, then multiply by a kajillion future simulated lives or whatever.) most of the philosophers who research this stuff, and most of the people involved in Effective Altruism, are part of the same niche subculture. i don't think this is a grift, I think they are quite honest in their convictions. but it makes me go "hmmmmm".