I've often thought that if you want to represent a probabilistic world model, with nodes that represent physical objects in space-time (and planned-future space

I've often thought that if you want to represent a probabilistic world model, with nodes that represent physical objects in space-time (and planned-future space-time) and our level of certainty about their relationships to one another... you'd do that outside an LLM's token stream. You could, in theory, represent that model as a linear stream of tokens, and provide it as context to an LLM directly. It would be an absurdly wasteful number of tokens, at minimum, and the attention-esque algorithm for how someone might "skim" that model given a structured query would be very different from how we skim over text, or image patches, or other things we represent in the token stream of typical multi-modal LLMs. But could it instead be something that we provide as a tool to LLMs, and use an LLM as the reasoning system to generate structured commands that interact with it? I would wager that anyone who's read a book, drawn a map of the fantasy world within, and argued about that map's validity on the internet, would consider this a viable path. At the end of the day, I think that the notion of a "pure LLM" is somewhat pedantic, because the very term LLM encapsulates our capability of "gluing" unstructured text to other arbitrary tools and models. Did we ever expect to tie our hands behind our back and make it so those arbitrary tools and models aren't allowed to maintain state? And if they can maintain state, then they can maintain the world model, and let the LLM apply the "bitter lesson" that compute always wins, on how to best interact with and update that state.