> A perfectly mediocre essay. The author rightly draws attention to text. LLMs showed they can do the classical NLP tasks and more: summarise, translate, answ

> A perfectly mediocre essay. The author rightly draws attention to text. LLMs showed they can do the classical NLP tasks and more: summarise, translate, answer questions, play a role, brainstorm ideas, write code, execute a step by step procedure, the list is unbounded. It's the new programming language. All these abilities emerged from a random init + text. Guess what was the important bit here? Text. It's not the architecture, we know many different architectures and they all learn, some better than others, but they all do. Text is the magic dust that turns a random init into a bingChat with overactive emotional activity. Here I think the author made us a big service in emphasising the text corpus. We were lost into a-priori thinking like "it's just matrix multiplication", "it's just a probability distribution predictor over the next token". But we forgot the real hero. The interesting thing about words is that they are perceptions, they represent a way to perceive the world. But they are also actions. Being both at the same time, perception and action, that makes for an interesting reinforcement learning setup, and one with huge training data. Maybe text is all you need, it is a special kind of data, it's our mind-data.