I think part of the problem is that LLMs seem to be quite effective at producing messages adhering to ulterior motives, catch attention, reinforce emotions etc.

I think part of the problem is that LLMs seem to be quite effective at producing messages adhering to ulterior motives, catch attention, reinforce emotions etc. The GPT-4 release documentation has some examples of this in its addendum. ChatGPT also seems to be good at writing advertisements. Without the strong guardrails, I wouldn't bet on one or two persons instructimg a GPT-4-scale model perfoming worse at manipulating debates than 10 or 100 humans without AI.