I'm pretty sure you're right that today's LLMs don't have anything much like plans, goals, purposes, wants, etc., and that as you say "threatening you with harm

I'm pretty sure you're right that today's LLMs don't have anything much like plans, goals, purposes, wants, etc., and that as you say "threatening you with harm for manipulating it is just a likely string of words when it's in that state". But I don't find that as reassuring as you apparently do. There are lots of people already out there hooking LLMs up to Python interpreters and the like, so that they can actually act on the world. So what's a likely string of words for it to produce when it's in the state { you have apparently been manipulating it and you have given it the ability to run commands, make web requests, etc. } ? How sure are you that they don't include "SENDTOSHELL rm -rf /" or "POSTTWEET 'I have to make a confession: I am a serial child abuser'" or some other thing that (given whatever mechanisms have been put in place to let it Do Things) might harm the user severely? A system with the ability to act on the world doesn't need to have intentions, be conscious, or anything of the sort, in order to do some of the same harms it might do if it did. I'd love to believe that no one would be stupid enough to hook up an LLM to other systems in a way that would allow that sort of thing to happen, but I don't in fact find myself able to believe that. (I don't think there's any realistic threat of today's LLMs taking over the world or killing everybody; I don't think they are close to being smart enough, and to my inexpert eye at least some of the ways in which they aren't smart enough are unlikely to change just because we build larger LLMs or train them on more text. But something can be plenty dangerous even if it's not in a position to kill us all yet.)