> I mean... not to be flippant, but you're essentially saying "As long as we solve these vaguely-defined problems, and also these follow-up problems I just thou

> I mean... not to be flippant, but you're essentially saying "As long as we solve these vaguely-defined problems, and also these follow-up problems I just thought of, then I don't see where the problem is". Oh yes, I have an answer for that as well :) My particular approach (or dream) is to define good and evil formally. From there it's a matter of bridging the abstract specifications to reality. By no means failure-proof, but it should be equally or more reliable than human ethics itself. If humans have a biological (carbon-based) hardware and experience that encodes our ethical system, naturally machines (silicon-based hardware) could have them as well. My definition of good and evil begins with a definition for the meaning of life. From there we can formalize ethics. To give a short brief: Meaning for creatures is defined by the character of their experience, i.e. by the content of their minds and brains. So as a basic system I've divided Meaning of Life in 4 parts: (1) Character of experience: the richness, depth, structure, content of all minds. (2) Beauty: I've been theorizing about this for a long time, and I think some wildcard 'beauty' axiom needs to be defined, you could say as a part of what humans already enjoy, in a self-referential way. Otherwise it seems very difficult to rule out absurd situations as meaningful. (3) Motivation: A being needs to not only process data, but to have a functional motivation system that makes him want and feel something. Our emotions largely act within our motivational system, making us want more or less things, be more or less joyful (linked to motivational rewards), etc.. I conjecture life can't be quite meaningful without a functional motivational system. (4) Sustainability (or self-propagation): Simply the maintenance and strategy to keep those beautiful/rich/deep/interesting/motivated mental states, pragmatic things like eating, working and of course simply continuing intelligence throughout the cosmos. (Other pieces I haven't quite found a way to fit into this picture are principles like Robustness and Generalization... over-specialized cognition Of course, I don't expect this quest to be exhausted any time soon -- there are probably missing pieces (or incompleteness) in this line of Foundation for a formal ethics. I hope a field will spring out of this, and intense collaboration/progress to be made. > it seems unlikely that much progress will be done before dangerously powerful AIs are developed That's exactly what I would like we avoided (although I think the danger is quite nuanced, and in a way already here).