I mostly agree with what you said, and I'm also skeptical enough about LLMs being a path towards AGI, even if they are really impressive. But there's something

I mostly agree with what you said, and I'm also skeptical enough about LLMs being a path towards AGI, even if they are really impressive. But there's something to say regarding these things not getting ideas or self-starting. The way these "chat" models work reminds me of internal dialogue; they start with a prompt, but then they could proceed forever from there, without any additional prompts. Whatever the initial input was, a session like this could potentially converge on something completely unrelated to the intention of whoever started that, and this convergence could be interpreted as "getting ideas" in terms of the internal representation of the LLM. Now, from an external point of view, the model would still just be producing text. But if the text was connected with the external world with some kind of feedback loop, eg some people actually acting on what they interpret the text as saying and then reporting back, then the specific session/context could potentially have agency. Would such a system be able to do anything significant or dangerous? Intuitively, I don't think that would be the case right now, but it wouldn't be technically impossible; it would all depend on the emergent properties of the training+feedback system, which nobody can predict as far as I know.