This article describes an experiment that uses a very simple "world" with three states. Two of the states emit "0" and "1", respectively, and third is a little

This article describes an experiment that uses a very simple "world" with three states. Two of the states emit "0" and "1", respectively, and third is a little more tricky - it emits either "0" or "1" randomly. The transformer attempts to predict the next token, and they show that it also attempts to guess the current (hidden) state. So far, so good. A natural interpretation of this experiment is that there is a "timeline" identical to the stream of tokens - that is, the time coordinate is given by the current offset into the stream. Each token is evidence of the world's current state at the time corresponding to that offset. But there's no particular reason to call that axis "time." We could just as easily treat it as an x-coordinate, and there's no time at all. The next token is just the one to the right. Real stories often don't have such a simple mapping. The offset into the text stream is a different dimension than the story's timeline. A "world model" worth its salt would have to attempt to put the various events described within the text stream into some kind of partial order, using various clues about how the events are described. (For a simple experiment, the events could be timestamped to make it easier.) It would be interesting to know how transformers represent the "current time" and how that relates to other relevant times in a story. I wonder if there's any research about that yet? Also, could they learn more sophisticated representations from tricky stories involving time travel?