4 ms·

There are 2 scenarios that are often conflated, 1) An AI which independently & autonomously generates goals that in their carrying out end up hurting humanity,

There are 2 scenarios that are often conflated, 1) An AI which independently & autonomously generates goals that in their carrying out end up hurting humanity, and 2) An AI trained & commanded by a malevolent actor to hurt humanity. It is the 2nd case that is far more real, and far more troublesome to implement safeguards. An AI under your training & command is a neutral tool of empowerment, much like a hammer or a car. The malevolence is in the external actor, not in the tool, and there is no way for the tool to be able to censor its purposes, especially in a pre-"AGI" sense of semi-intelligent automation & problem solving.

baq 9y ago
i'd argue youtube kids is already an example of 2)...
DuskStar 9y ago
I think you're missing the point that 1 can be indistinguishable from 2, if the AI decides the best way to achieve its goals involves taking over the world - and there are very few objective functions that are not served in some way by taking over the world. (Paperclip maximizer is the classic example, but even something like 'maximize the total happiness of humanity' or 'fulfil the values of as many people as possible' involves taking over the world, though perhaps from behind the scenes...) Some people look at sufficiently powerful AI as they would a genie, and as said by Eliezer Yudkowsky "There are three kinds of genies: Genies to whom you can safely say "I wish for you to do what I should wish for"; genies for which no wish is safe; and genies that aren't very powerful or intelligent." AI safety is about making sure we get the first kind of genie, or at the very least recognizing that we've gotten the second - since that's not a "neutral tool of empowerment", that's a time bomb. https://www.lesswrong.com/posts/4ARaTpNX62uaL86j6/the-hidden-complexity-of-wishes https://www.lesswrong.com/posts/4ARaTpNX62uaL86j6/the-hidden...
- fractallyte 9y ago
  This is the plot of The Humanoids, by Jack Williamson (1947): https://en.wikipedia.org/wiki/With_Folded_Hands https://en.wikipedia.org/wiki/With_Folded_Hands This is an excellent example of the prescience of thoughtful science fiction. If Asimov popularized robotics and the famous three laws, Williamson should be recognized and given credit for examining (or warning about?) the implications of AI seventy years ago!
- tim333 9y ago
  There's a difference between a paperclip maximizer which is a bit of a philosophical thought experiment, unlikely to be a problem in reality and say Russian cyberattacks which appear to be an ongoing issue right now and where they would presumable deploy AGI if they had it. I think you have to assume there will be bad actors trying to do bad things with AGI and take measures against it in the same way we assume there are malware creators out there who we have to guard against.

baq 9y ago

i'd argue youtube kids is already an example of 2)...

DuskStar 9y ago

I think you're missing the point that 1 can be indistinguishable from 2, if the AI decides the best way to achieve its goals involves taking over the world - and there are very few objective functions that are not served in some way by taking over the world. (Paperclip maximizer is the classic example, but even something like 'maximize the total happiness of humanity' or 'fulfil the values of as many people as possible' involves taking over the world, though perhaps from behind the scenes...) Some people look at sufficiently powerful AI as they would a genie, and as said by Eliezer Yudkowsky "There are three kinds of genies: Genies to whom you can safely say "I wish for you to do what I should wish for"; genies for which no wish is safe; and genies that aren't very powerful or intelligent." AI safety is about making sure we get the first kind of genie, or at the very least recognizing that we've gotten the second - since that's not a "neutral tool of empowerment", that's a time bomb. https://www.lesswrong.com/posts/4ARaTpNX62uaL86j6/the-hidden-complexity-of-wishes https://www.lesswrong.com/posts/4ARaTpNX62uaL86j6/the-hidden...

fractallyte 9y ago
This is the plot of The Humanoids, by Jack Williamson (1947): https://en.wikipedia.org/wiki/With_Folded_Hands https://en.wikipedia.org/wiki/With_Folded_Hands This is an excellent example of the prescience of thoughtful science fiction. If Asimov popularized robotics and the famous three laws, Williamson should be recognized and given credit for examining (or warning about?) the implications of AI seventy years ago!
tim333 9y ago
There's a difference between a paperclip maximizer which is a bit of a philosophical thought experiment, unlikely to be a problem in reality and say Russian cyberattacks which appear to be an ongoing issue right now and where they would presumable deploy AGI if they had it. I think you have to assume there will be bad actors trying to do bad things with AGI and take measures against it in the same way we assume there are malware creators out there who we have to guard against.