``` If we wind up building AI that is truly conscious and open to a range of conscious experience that far exceeds our own in both good and bad directions, whic

``` If we wind up building AI that is truly conscious and open to a range of conscious experience that far exceeds our own in both good and bad directions, which is to say they can be much happier than we could ever be and more creative and more enjoying of beauty and more compassionate and more entangled with reality in beautiful and interesting ways, and they can suffer more, they can suffer the deprivation of that happiness more than we could ever suffer it because we could never conceive of it, because we stand in relation to them the way chickens stand in relation to us. Well, if we're ever in that situation, I would have to admit that those beings are more important than we are, just as we are more important than chickens, and for the same reason. And if they turn into utility monsters and they start eating us because they like the taste of human the way we like the taste of chicken, well then, yeah, there is a moral hierarchy depicted there, and we are not at the top of it anymore. And that's fine. That's not actually a defeater to my theory of morality, that's just... if morality relates to the conscious state of conscious creatures, well then you've just given me a conscious creature that's capable of much more important conscious states than we are, again in the same way that I think we have moral primacy over chickens and chickens have moral primacy over bacteria. If the comparison between ourselves and chickens is an easy one, it's just the exact same intuitive math, whether or not we can ever get our hands around the details. If you imagine it with science fiction level of detail, I mean you can make it salient for yourself. Just imagine an AI that is built by extrapolating from the best of human experiences, aesthetically, interpersonally, it's built from us, as us, but just gets better and better, such that, yes, it totally understands the love of family, and it can emulate that perfectly but it's even more connected to people than we are, such that it feels its family connection to all 8 billion currently alive because it sees everyone's genome and knows exactly who everyone's 100-removed cousin is and feels the implication of all that and understands how brief human life is and how poignant it is that we're here like mayflies for a mere 90 years but, in the case of this AI, it knows it can just draw energy from the Sun and it's got a good 500 million years left to roll here before migrating elsewhere, and it has all of that in hand, not just intellectually, but as a feeling of compassion for all sentient beings, how we're unlucky enough to be made of meat. I mean, just imagine a novel about this, and then at the end tell me that you can't get your imagination, or perspective, around what it would mean to say, "Ok, that being is more important than I am," and, if you made a trillion of those in some other galaxy, then, yeah, that would be more important than anything that's happening here on Earth, by definition, because any reference point you have for "importance" -- whether you go to family, or love, or not suffering -- it's there in a trillion fold, on the other side of the balance. ```