If clustering has worked on what amounts to basically the entire world of information things get a bit fuzzy though. I don't suppose you are technically incorre

If clustering has worked on what amounts to basically the entire world of information things get a bit fuzzy though. I don't suppose you are technically incorrect, it's just that these words lose practical meaning when we talk about models that encode tens or even hundreds of billions of parameters. Predicting the "next token" requires an "internal model of the world". It might not be how we do it, but without something that acts like it I'd be very interested in how you think it comes up with its predictions. Let's say it needs to continue a short story about a detective. The detective says at the end: "[...] I have seen every clue and thought of every scenario. I will tell you who the killer is:". Good luck continuing that with any sort of accuracy if you don't have some abstract map of how "people" act. You can see how I can think of a lot of examples that require something that acts as a model of the "world". There's a definite structure and pattern to everything we do. This (to an LLM) hidden context gives rise to the words we write. To re-invent them, like it has to do, it must basically conjure up all this hidden state. I'm not saying it gets it right, I'm just saying that there is no other way than to model the world behind the text to even get into ballpark-right territory.