The training process forces it to optimise predicting the next token, across an extremely large and diverse set of tokens. To do this, it has access to a very l

The training process forces it to optimise predicting the next token, across an extremely large and diverse set of tokens. To do this, it has access to a very large and layered neural network, "the weights", after the particular property of this network that changes during training. We largely do not know what that neural network does in order to successfully predict the next token. We know how it learned in an obvious sense, it adjusts weights randomly on a pass and then evaluates if the loss is lower, preserving or rejecting weight adjustments as necessary using gradient descent. We know in a general sense "the neural net takes information from previous tokens and transforms them into a prediction for the next token". But we largely don't know how. This is where the controversy over whether it engages in reasoning or logic comes from; it clearly does, we can see it in how it responds to certain questions, but it is not obvious where in this process anything like a logic engine could fit in. Some convincing research has suggested that the neural network generates a world model ("models the world" might be easier to understand grammatically) based on the information in the input data (the internet), which it then uses during inference to evaluate questions, with different parts of the model activating for different tasks, like recalling a memorised fact ("When was the Grand Canyon named?"), or answering a logical puzzle like the ones with Alex and Bob. A hypothesis is that the process of taking a large neural net and forcing it to learn how to predict the next token on extremely large datasets with lots of compute gives it the ability to create "things" within itself, subnetworks or little logic engines or coarse emulations of various intelligent entities. After fine-tuning and all the other housekeeping, if you ask it a question it will figure out which of its "things" can answer that type of question, hand it to that "thing", and the "thing" uses both memorised information and other "things" to come up with an answer. If this hypothesis is true, there is no contradiction between "it predicts the next token" and "it reasons logically and relies on its memory to answer the questions it is asked". The full sentence would be "It reasons logically and relies on its memory in order to answer the questions it is asked, by predicting one token at a time." I have not seen another hypothesis that adequately explains to me how it is possible for the current crop of LLMs to correctly answer novel logic questions. A ~good challenge to this hypothesis is that there are still problems that LLMs can't solve, like correctly formatted novel Winograd schemas, and it's not clear why this hypothesis should work for the things it can do but not for them. My answer to that challenge would probably be: build a bigger system and see if that can solve those other problems too. Scale seems to be the overwhelmingly dominant factor in performance for these things.