4 ms·
> AI makes it even easier to spin up fake personas to pretend to be human and advocate It's not only that. AI enables a never-before-seen level of individual t
by cornholio 1y ago
> AI makes it even easier to spin up fake personas to pretend to be human and advocate
It's not only that. AI enables a never-before-seen level of individual targeting for political and commercial actors, campaigns of behavioral modification and radicalization, to the point where the entire intelectual foundations of democracy become questionable.
When power actors addressed the people in traditional media they could send a single message that was tailored to maximize effect, but necessarily needed to be addressed to the common man. The explosion of internet fragmented the media space, but we're still talking about unitary publications, say, an opinion piece presented identically to all online subscribers of a certain publication, with narratives targeting broad swaths of the population: young urban males, conservative retirees and so on. Cambridge Analytica disrupted that model, allowing targeting based on individual profiling, A/B testing to see what kind of content works best on people with certain proclivities etc.; but again, the decisions were relatively low complexity and automatic.
Now imagine each individual has a dedicated GPT-5 level agent following him around across devices and media, that operates 24/7 with the singular task of influencing his opinion, convince him to join a cause, plunge him into depression, buy something, or whatever else the power actor needs from that individual. This agent not only has an excelent profile of his target and can generate videos, fake personas etc. as necessary, but also has a near expert level competence in things like psychology, persuasion and manipulation. It doesn't just push narratives, its tasked with convincing you and isolating you from whatever external influence threatens that goal, and it reasons towards that goal with near expert level accuracy. Would 99% of the population resist such a brainwashing machine? Would you?
This is the type of agent Facebook and Twitter/x are striving towards. It's a world where people no longer have common understanding of a shared social reality, and collaboration towards keeping Power in check becomes fundamentally impossible. It's orwellian to a degree even Orwell didn't imagine.
- ants_everywhere 1y agoYeah for sure. But also Orwell was extrapolating from the reality on the ground in communist countries. The scale achieved there was also massive. Your neighbors would turn you in if you criticized the people in power, etc. This is also the reason the Soviets were ahead of the US in trying to train people like dogs (Pavlov's research) and why the communist forces ran brainwashing experiments on American POWs during the Korean War. If you look at what countries were willing to do back then it should make you concerned about what's coming. For personalized targeting with psychological warfare, perhaps the closest analog is socialist East Germany's Zersetung https://en.wikipedia.org/wiki/Zersetzung https://en.wikipedia.org/wiki/Zersetzung The main difference you get with LLMs is that it's cheaper to achieve these same goals Orwell was concerned about. You no longer have to make explicit and credible threats of violence. It's also easier to reach people in democracies and convince them that democracy is bad etc. So for the sorts of concerns Orwell had, I think we're already seeing that. And Deepseek is one weapon in that war, since it has to comply with the Chinese regulations that LLMs must spread socialist core values. How would you resist? I think a necessary precondition is that people continue to champion the importance of democracy and freedom of thought.
- extropic-engine 1y agohave you looked at the US lately? if you're concerned for democracy i don't think deepseek is the one you need to be most worried about
- ants_everywhere 1y agoIt's the one LLM we know of that is specifically trained to be opposed to democracy. Can you say more about why that doesn't concern you?
- djeastm 1y agoI would throw Grok in there, too, but I suppose that might get us into a political discussion...
- ants_everywhere 1y agoI don't know anything about Grok, but AFAIK they have different problems. Grok lacks alignment to push back against users asking for bad things, like advocating for hurting people. But Deepseek is trained to manipulate the user into wanting bad things like hurting people.
- spencerflem 1y agoI disagree that Grok, manipulated by Elon Musk, does not push ideas that hurt people
- johnecheck 1y agoYeah. We need better tools that empower individuals to think independently and we need them now. Education is part of the answer, but I fear it isn't enough.