Not only the scale but also the multi-agent nature of the problem makes it hard to predict. As an example, humans killing each other is an ancient problem. But

Not only the scale but also the multi-agent nature of the problem makes it hard to predict. As an example, humans killing each other is an ancient problem. But humans killing each other at scale with nuclear weapons is a much different problem. We are able to keep that threat at bay by the virtue of only nation states having the power of killing at this scale; number of agents that need to coordinate not defecting (that understand mutually assured destruction) is a handful. Now imagine if some new technology made it possible to make nuclear weapons with materials found in a kitchen. Then we have a coordination problem that requires planet level cooperation with billions of agents. It only takes one defecting to cause disproportionate harm. By the same token, information warfare is not a new problem. Lying is an ancient invention. News outlets being biased makes it easy to guess those biases and somewhat balance them out. We expect states to engage in some degree of propaganda. But human created but AI curated content (i.e. recommendation algorithms) is already driving us a bit nuts; we are having a hard time making a cohorent narrative of what’s going on. AI created and curated content intermixed with human content will completely destroy our sensemaking capacity for the benefit of a few who can deploy this technology. Imagine a per-voter propaganda model that drives AI created social media interactions with them, that can precisely tell what issues would you need to change your mind on to make you change your vote, which could be polar opposite to what the next voter is going to hear. Imagine this was deployed by multiple nations whenever there is an election in a country. This would still at the level of nation-states having nuclear weapons. I can’t imagine what the consequences of a more diffuse access to this tech would look like.