I think I mostly agree with you, but I think this framing is a bit misleading. On autocomplete, I'll just lazily quote the relevant part from my post: It’s com

I think I mostly agree with you, but I think this framing is a bit misleading. On autocomplete, I'll just lazily quote the relevant part from my post: It’s completely true that LLMs are trained on next-token prediction (although some, like ChatGPT, are then additionally trained using reinforcement learning with human feedback). It’s also completely true that this fact profoundly influences the texts they generate. So I don’t think it’s unreasonable to call LLMs autocomplete engines or to emphasise next-token prediction. But I think it’s subtly misleading: - Though LLMs were trained to optimise success on next-token prediction, that is not necessarily what they do. We don’t know what it is they do. The training process reinforces behaviours/heuristics in the model that tend to cause it to make better next-token predictions on in-distribution data. This does not mean that those behaviours/heuristics are fundamentally “about” optimising next-token prediction, especially when the model encounters out-of-distribution data. - The usual example here is human evolution. Humans were shaped by a process that optimised for reproductive fitness. This gave us a bundle of drives such as family kinship, prestige and sexual pleasure – drives that aren’t fundamentally about optimising for reproduction, which becomes evident as we enter a new environment – one with contraceptives, say. - Optimising for a task for which intelligence is useful encourages the optimised thing to become more intelligent. Sam Altman gave expression to this last week when he wrote, “Language models just being programmed to try to predict the next word is true, but it’s not the dunk some people think it is. Animals, including us, are just programmed to try to survive and reproduce, and yet amazingly complex and beautiful stuff comes from it.” - The forms of intelligence that are useful in doing next-token prediction are different from those that are useful in human reproduction, but I think there’s a considerable overlap, as (1) some fundamental abilities, for example using and applying concepts, just seem very broadly useful and (2) the data LLMs are trained on are written by humans, for humans and often about humans and things that matter to us. I think the "they're just autocomplete" take also hides other properties of LLMs, like them seeming to (as mentioned in another comment, and in the post) contain and use world models, and being able to learn general algorithms.