I think that AGI is a poorly understood concept and whatever it would signify, we're a couple of paradigm shifts away from it. One thing that bothers me is how

I think that AGI is a poorly understood concept and whatever it would signify, we're a couple of paradigm shifts away from it. One thing that bothers me is how there is little concern of legal personhood here, apart from popular action film style "rebellion" fears. (Let's remind ourselves of the 8 billion human level intelligences we're having right now, mostly not wreaking much havoc individually). Let's say that some organization builds an AGI that's conscious and capable of human-like thoughts. There's an argument to be made that at that moment, if we don't immediately grant the AGI personal and political rights of citizens, this organization is essentially owning slaves. Which is contrary to the rules of our societies and never ends well. It seems likely to me that, barring some immediate and fantastical gains from AGI that would allow the organization to "break" the society, it would be soon forfeited as property with no compensation. Which makes a commercial venture to build AGI a dubious proposition.