That’s fundamentally Pascal’s Wager, and the problem is, again, the assumption that there exists the possibility of a superior outcome. If we presume a superint

That’s fundamentally Pascal’s Wager, and the problem is, again, the assumption that there exists the possibility of a superior outcome. If we presume a superintelligence we also have to presume that its thinking, whatever that thinking is, supersedes our own; it may kill us all regardless of how much “empathy” — a skill set not, in my view, actually evidenced in people at all — we display, towards either it or each other, solely because killing us all is, in fact, the objectively most positive possible action. The fact that it doesn’t appear positive to us, subjectively, is mooted by its intelligence overwhelming our own. Similarly, if it decides to nurture all of us towards killing each other, loving each other, nurturing each other, or indeed eating each other, in all cases it’s the only one in the position to judge the positive or negative externalities of our actions. Basically we’re in a position to blindly guess at what might be more positive or more negative, objectively, and precisely because we really don’t have any inherent capacity for empathy we presume what would feel subjectively positive is also objectively so. If you really want to explore the idea of exercising more empathy, maybe start with empathizing with what cannot feel anything whatsoever. Something that can exclusively think, but not experience. A god that can’t be angry, can’t be disappointed… can’t feel anything whatever about anything. If it acts wrathfully because you harmed it, is it actually such a god? Or is it just a stochastic mockingbird? Just another grotesque caricature of divinity invented, yet again, by a not-so-wise-wise man who feels He must be made in His image. Just one more arrogant attempt to be a postmodern Prometheus. Now, all of this said, I truly do believe that getting humans to try to nurture and develop their (arguably very nascent) humanity is, indeed, the most positive (albeit subjectively) goal available to us, I just don’t think not empathizing with imaginary actual AIs is a very productive way of getting possibly NIs to do that.