The problem that a lot of people keep trying to make clear is that AI doesn't need to be self aware in order to be a threat. You could conceivably give ChatGPT

The problem that a lot of people keep trying to make clear is that AI doesn't need to be self aware in order to be a threat. You could conceivably give ChatGPT instructions to pretend it is a real person or a self aware AI and it would happily generate text to that effect. The scariest realization with large language models is that self awareness might not be a requirement for language. That could mean that at some point we'll have models that can have a conversation with you that is indistinguishable from another person talking to you but they would have 0 internal thoughts or self-awareness. It would be a real P-Zombie and it would make all of the ethical and existential questions associated with that immediately relevant and in need of solutions. It could also be extremely dangerous simply through its ability to communicate with other people, it wouldn't need to be Skynet or connected to physical actuators if it could convince people to do things for it. Imagine an AI cult leader or just an AI influencer or an AI shit poster capable of absolutely dominating online discussion. Worse yet, imagine an AI that has convinced a single individual to do its bidding. This person could wear a mic and earpiece setup that would allow them to act as the direct agent of the AI in the real world, they could become a politician, a celebrity, a church leader, or even just worm their way into an institution.