Good point. I've thought for a while now that one of the big risks of present AI chatbots is that they can endlessly validate any arbitrary thoughts of an emoti

Good point. I've thought for a while now that one of the big risks of present AI chatbots is that they can endlessly validate any arbitrary thoughts of an emotionally unstable human operator, creating a feedback loop that results in a strong irrational conviction, that might even be harmful to the operator or to others. (See: Lemoine and LaMBDA, or the person you linked to.) So I guess the risk there would be, the AI is accessible by the public, some fraction of the public "radicalizes" itself in a similar direction from talking with the AI too much, these people form a coherent movement, and that movement becomes powerful enough to take over the world. So then the question becomes, just how plausible is it for grassroots rebellions so formed to succeed against the authorities in real life, as opposed to fiction? The fundamental issue here is that the attack can occur through untargeted manipulation of vulnerable people, as opposed to the targeted manipulation of specific people in power (which I suspect near-future AI models will still be incapable of). The obvious defense would be to only allow AIs to be prompted by a sufficiently-large committee, alongside some social machinery in place to make sure the committee isn't all colluding as part of a cult. But I doubt many of the AI-risk people would ever accept that, operating under the shockingly common "powerful aligned AI or bust" model.