In and of themselves, programs want nothing. Think of "hello world". It wants nothing. It desires neither life, nor death, or anything else. What needs to be ca

In and of themselves, programs want nothing. Think of "hello world". It wants nothing. It desires neither life, nor death, or anything else. What needs to be carefully considered are the goals given strong AI at initialization. Consider the paperclip maximizer: http://wiki.lesswrong.com/wiki/Paperclip_maximizer http://wiki.lesswrong.com/wiki/Paperclip_maximizer The wording here understates the problem--the program might start with a goal as benign as "collect paperclips" as opposed to "fill the universe with as many paperclips as possible" but the outcome could be similar. Given the ease with which programs can be copied or modified (and mistakes made), I can't see many futures filled with strong AI's that end positively for us, chiefly because of the asymmetry in difficulty of destroying civilization vs. protecting it from all possible dangers.