You raise good points. In fact, that is the very problem of so-called "Friendly" AI. Note the capital F. It's a confusing term, I know, but the basic idea is an

You raise good points. In fact, that is the very problem of so-called "Friendly" AI. Note the capital F. It's a confusing term, I know, but the basic idea is an AI that "does what we want", in the most general sense. The idea behind FAi is the exact idea you present here. The point is that it is a bad idea to just throw together an AI and hope that it is friendly (lower case f), but rather the big important problem of our time is to generate a mathematical model of Friendliness, so that we can explicitly define to the AI how we don't want all the bad things you mentioned or any other besides. [Edit] Here are some relevant links: http://intelligence.org/files/CEV.pdf http://intelligence.org/files/CEV.pdf http://commonsenseatheism.com/?p=14047 http://commonsenseatheism.com/?p=14047 http://www.acceleratingfuture.com/michael/blog/2009/08/a-nice-and-meaty-introduction-to-friendly-ai/ http://www.acceleratingfuture.com/michael/blog/2009/08/a-nic...