Here's how I would summarize this (interesting) paper (please correct as appropriate). 1. Assume that consciousness is an illusion of the brain (Illusionism).

Here's how I would summarize this (interesting) paper (please correct as appropriate). 1. Assume that consciousness is an illusion of the brain (Illusionism). That is, when we see the color red, the brain creates qualia (the feeling of seeing), but it's just an illusion.[a] 2. Why is the brain like that? The Attention Schema Theory (AST) suggests that this helps animals to focus on important inputs. The brain turns certain inputs into qualia (sensations), while ignoring others, to help us focus. Consciousness is kind of a like a blackboard on which certain inputs are placed, which helps higher planning functions to make decisions. 3. Once we have this sense of self, we can infer that other beings also have a sense of self and are able to form social bonds. Moreover, we evolve a morality based on having a sense of self. We believe inflicting pain is morally bad because we don't like feeling pain and can empathize.[b] 4. If Illusionism and AST is true, then it is likely that alien intelligence also evolved in the same way (because of convergent evolution). Therefore, alien intelligence might also have consciousness and might have also developed a moral framework based on consciousness. 5. But what if Artificial General Intelligence is possible? If AGI is possible, then it is likely that alien species quickly transition to a post-biologic state. Even if it takes us another 1,000 years to develop AGI, that's nothing in the lifetime of a species. Any alien intelligence we encounter is likely to be millions of years ahead of us (evolutionarily) and therefore has already developed AGI.[c] 6. Moreover, just as AST is useful in us to efficiently use brain resources, it should be useful in AGI. Thus, AGI might be programmed with (or eventually evolve) consciousness. 7. But such consciousness might be very different from biologically created consciousness and that would lead to misalignment. For example, an AGI might be nihilistic and not think consciousness is special. From their perspective, killing a human would be no different than disassembling a laptop. --------------- [a]: This is in contrast to theories that say that qualia/consciousness is some unique (and unknown) physical process. For example, Roger Penrose thinks qualia is some kind of quantum mechanical effects (I personally think that's bordering on pseudoscience, but what do I know). [b]: Consciousness is intertwined with free will here. You can't have free will without consciousness (cars don't have free will) and if you have consciousness, you have free will (at least as far as humans are concerned). And free will is the basis of all morality. You can't hold someone responsible unless they have free will. And thus, you can't hold someone responsible unless they have consciousness. [c]: Alien civilizations can't be millions of years behind us because then they wouldn't have technology. And they can't be equally advanced because in a universe that is 12 billion years old, having two civilizations the same age would be an awful coincidence.