> ... If the algorithm was so intelligent as to know how to go from manufacturing paper clips out of supplied resources to deciding how to get more of the neces

> ... If the algorithm was so intelligent as to know how to go from manufacturing paper clips out of supplied resources to deciding how to get more of the necessary resources by killing humans, it could probably figure out that this is pointless. This is an interesting point. I'm not convinced that it could decide or figure out that the operation is pointless (for reasons I'll explain below), and I feel like it is perhaps taking the paperclip example to an unnatural extreme. Really, the crux of the matter is that any intelligence agent at human level would have the intelligence and problem solving skills we do without the moral and social constraints that we do. Any problem it decides to solve or any task it's given to complete will be performed with the same ruthless efficiency you'd expect a machine to have. This is why it might not be able to figure out that the operation is 'pointless' or 'wrong'. If you give it an objective but don't provide first order or axiomatic values for human life or the non-substitutability of people, then you'll get behaviors which you would consider evil. Consider the following: "Person arm stuck in assembly line. Halt assembly line for 1hr, lower-bound for total cost: $1,200,000. Continue assembly, upper bound for injury settlement cost: $1,000,000. Decision: continue assembly." There's no implicit consideration of human worth that people have because human morality is relative. In general, we have a similar set of base parameters instilled by evolution (cute things and things like us are good -- don't hurt or kill them), but we can't prove from any mathematical axioms that these are 'right'. This is why people get scared -- if you program a computer to maximize the profits of a company and to do so at any cost, a human-level AI will spend an eternity doing just that, even if it means the complete subjugation of humanity. (Since we can't eliminate humanity otherwise currency ceases to exist and economic collapse will hurt the bottom line.) I would wager (but can't prove) that a study of abused people, child soldiers, and sociopaths would yield interesting parallel psychological profiles. These are people (human level) with a set of axioms which are distorted by artificial means. There's still the built-in problem of the human component adding "Don't hurt the cute things," in most cases but nonetheless.