3 ms·
> Our results demonstrate that the power of language models can be attributed, to a great extent, to the auto regressive next token training scheme, and not nec
by visarga 3y ago
> Our results demonstrate that the power of language models can be attributed, to a great extent, to the auto regressive next token training scheme, and not necessarily to a particular choice of architecture.
I would have hoped they would attribute LLM success to the structure of language itself. As the authors say, even small linear models can approximate CoT and solve complex tasks. So it's not the model. It's the data.
Analogously, humans have very different brains when looking at low level, but brains still learn the same language and skills about as good as any other brain. It's not the brain or neural net (the models) but the data that shapes them to become smart.
This insight has consequences on how we view training data and where to focus our work to improve AI and human brains - improve language, ideas & chains of thought. This resonates with recent discoveries in fine-tuning and training small models like phi-1 and phi-1.5 who were trained on "textbook quality" data of high diversity.
- jacobsimon 3y agoSuper interesting - this almost sounds like evidence of linguistic determinism / relativism, i.e. the idea that our language influences how we perceive and think. Is that what you’re thinking too?
- Sharlin 3y agoI don't think that's what the GP is going for – more like that language is a way to general intelligence, and it doesn't matter what your "implementation" of language is. Just like there's an incredibly diverse range of models and formalisms proven to be Turing complete and as such, equivalent to each other, even though at first glance many of them look just as non-Turing-complete as an autoregressive next-token predictor looks non-AI-complete. (As all known human languages are Turing complete, every language should be completely equivalent in expressive power – only nuances are different and as such language doesn't affect thinking in any meaningful sense, and this seem to be corroborated by evidence.) ((On the other hand, it appears that individual people's brains indeed come up with "strategies" how to think – some think more verbally, others more visually, and yet others think in abstract, conceptual ways difficult to even put into words. For example, not nearly everyone has an "inner voice". Yet these strategies all appear to be approximately equivalent in their "thinking power".))
- mycall 3y agoDoes this imply a causation link between language skills and general intelligence? Rats are smart but have weak language skills, same with octopus afaik.
- visarga 3y agoYes, over long time spans language accumulates and transmits experience, so rats would be disadvantaged while humans got a huge boost. What I wanted to say originally is that we are riding on the shoulders of giants - the whole corpus of knowledge and concepts, ways to use them that have been discovered by previous generations, at great expense. AI and new born humans inherit this language heritage. That is why I would attribute to language most of our intelligence. Not all, because we adapt and contextualise, sometimes we stumble on a new thing, a new concept or piece of knowledge. But that is a rare phenomenon, we are 99% of the time contextually reusing the crystallised intelligence in language. We're like LLMs, where each token generated "visits" the synthesis of the whole human culture - the model weights - before being fully formed. Our thoughts travel the same path - they visit the model of human culture before being formed. If we lost our language and knowledge we'd have to redo the path again, over a long time, and pay the same price. In a sense, language is smarter than us - smarter than one human generation can accomplish.
- firtoz 3y agoIt certainly does. Speak to anyone who has learned more than a few languages :)