The mind design space is HUGE. (http://lesswrong.com/lw/rm/the_design_space_of_mindsingeneral/ http://lesswrong.com/lw/rm/the_design_space_of_mindsingenera...)

The mind design space is HUGE. (http://lesswrong.com/lw/rm/the_design_space_of_mindsingeneral/ http://lesswrong.com/lw/rm/the_design_space_of_mindsingenera...) The "default AI" is a program that we build. That's all we know. Most programs that we build do not properly represent human values. If they don't properly take those into account, then we lose things we care about. The AI that "optimizes our supply chain for paper clips" will, in the limiting case, consume humans and the environment and the earth and the sun in order to produce as many paperclips as possible and distribute them as widely as possible. A "default AI" will not care about its survival or the survival of its creators.