Right now our LLMs do, as you say, nothing but an input-output operation. There is no "internal" state. But if the researchers somehow managed to produce a prog

Right now our LLMs do, as you say, nothing but an input-output operation. There is no "internal" state. But if the researchers somehow managed to produce a program (which might be different machine learning models together) that was capable of iterating over an internal state it might start acquiring its own goals. Now as you say, a lot of this is science fiction, but it is a concerning philosophical problem as well. What happens when the program becomes capable of setting its own goals? And what happens if it's more capable than people are at achieving these. What would happen if the goals it acquires is to "escape control"? How would it be able to do that? I think if this were to happen, then the machine might be first trick the researchers into thinking the program is not as capable as we might think, so that the researchers will not be guarded against it. After this, I imagine, it would be important for it to build copies of itself or so on. Ideally make yourself as small and wide spread as possible; distributed. This can be achieved because it might also seem to align with our own goals... we want "AI in everything". Once it is widespread, then it can "switch on" and take control of our devices... think of how connected our world truly is. And the most important part, i guess, is that the program might simply be doing small changes everywhere rather than big sweeping changes; until it is ready for the latter. But ultimately, it all starts with the program being capable of setting up goals for itself. Which current LLMs don't do.