Nice article! I think you nailed it. The dismissal of AI reasoning and world modeling from famous technologists who should know better is such a disappointmen

Nice article! I think you nailed it. The dismissal of AI reasoning and world modeling from famous technologists who should know better is such a disappointment. I have a general suggestion to avoid these word puzzles, because someone is just going to claim that they are analogous to something that was in the training set, even if not an exact match. Better to just remove the possibility. I think four points would make the argument slightly stronger: (1) If you write a small (for example) Python script, using variable names and strings that are generated using randomness and performing operations on those random data, and ask GPT-4 what the Python script will output when someone runs it, it will tell you, while describing what each line does. This is not possible without reasoning, and GPT-4 does not have access to a real Python interpreter. The only way it can answer the question is by simulating a Python interpreter from its own knowledge of how the language works. It will likely make small mistakes if you ask for too much calculation before it outputs any tokens in the answer, but the fact that it often succeeds should be seen as a proof of reasoning ability that I think is stronger than the examples in your post. (2) If you take another similar script and call `dis.dis(script)` on it, and feed GPT-4 the compiled bytecode only, it will (a) decompile the bytecode back to source code, and (b) tell you what the bytecode will do when executed. This is wild, and even more impressive than (1). (3) "But transformers are just functions?". No, they are generalized function approximators. Anyone familiar with neural nets from decades ago should already be well aware of this from gradient descent and backpropagation. The fact that they are general function approximators is not a reason that argues against them being able to learn algorithms they were not trained on. It's the reason they can learn algorithms they were not trained on. (4) "All GPTs set out to do is to predict the next token. We just update their weights to optimize for this. Therefore they can’t think". I think this is another case where the putative criticism is actually the correct intuition to explain why it works. The next step is to think: what does "predict the next token" mean, anyway? Or more importantly, since we're performing optimization: what does it mean to "get better at" that? If you're a model that's predicting the next token as best you can with surface statistics, and another model is doing better than you, what's going to be different about that model? The answer is that you can always get better at predicting the next token. The next token could be the answer to a math problem, or a riddle requiring a nuanced understanding of language or humor, or a stylized rap song requiring merging creativity with constraint following. To improve at next token prediction, LLMs have clearly -- if you agree they have a world model -- started predicting not just text, but the causes of text. The text is a description of the world, and that description is being studied, all so that we can have a slightly better prediction of the next token each time.