3 ms·
The entire innovation (discovery?) of LLMs is that a good formula for the task of sequence completion turns out to also be a good formula for a wide range of AI
by nerdponx 3y ago
The entire innovation (discovery?) of LLMs is that a good formula for the task of sequence completion turns out to also be a good formula for a wide range of AI tasks. That emergent property is why language models are called language models.
- error_logic 3y agoThe usefulness is why the term is so widespread in familiarity but I think the term would have existed to describe the linguistic mapping even if they hadn't proven to have direct problem-solving capabilities.
- hackerlight 3y agoThat's an inductive bias, not an emergent property. The inductive bias of transformers is that they're good at integrating global context from different parts of a sequence without a particular bias towards recent time steps or localized regularities. And that happens to be a good fit for many (but not all) real-world sequence learning tasks. The "emergent property" aspect is when LLMs are good at a task at scale X*3 but were incompetent at scale X.
- nerdponx 3y agoMy point was that this particular inductive bias doesn't inherently beget a "language model".
- hackerlight 3y agoIt does beget a Transformer which we choose to call a language model when it's applied to language data