http://lesswrong.com/lw/ld/the_hidden_complexity_of_wishes/ http://lesswrong.com/lw/ld/the_hidden_complexity_of_wishes/ > To be a safe fulfiller of a wish, a g

http://lesswrong.com/lw/ld/the_hidden_complexity_of_wishes/ http://lesswrong.com/lw/ld/the_hidden_complexity_of_wishes/ > To be a safe fulfiller of a wish, a genie must share the same values that led you to make the wish. Otherwise the genie may not choose a path through time which leads to the destination you had in mind, or it may fail to exclude horrible side effects that would lead you to not even consider a plan in the first place. Wishes are leaky generalizations, derived from the huge but finite structure that is your entire morality; only by including this entire structure can you plug all the leaks. Humans can mostly differentiate between good and bad (ethics), but we don't know how we arrive at those conclusions (metaethics) because humans are terrible at introspection. Also, there's a ton of gray areas (e.g. the trolley problem). So rather than define all possible edge cases, it's probably less difficult to understand human decision-making from first principles and model our FAI accordingly.