4 ms·
We have had LLMs for much longer than 3 years.
by symfrog 4mo ago
We have had LLMs for much longer than 3 years.
- danielmarkbruce 4mo agoNo, we haven't, for any reasonable definition of L.
- wavemode 4mo agoOpenAI themselves must not have a "reasonable definition of L", then. Their own papers and press releases refer to GPT-2 (from 2019) as a "large language model". https://openai.com/index/better-language-models/ https://openai.com/index/better-language-models/
- danielmarkbruce 4mo agoYes, and 1.5 billion parameters meets no reasonable current definition of large. It would be considered a tiny language model. OpenAI themselves refer to their small/fast models as small models all over their documentation.
- wavemode 4mo agoThe term doesn't change its meaning because something new comes along. The point of the term "large" is to highlight the massive parameter count (compared to traditional statistical models, where having 1.5 billion parameters was basically unheard of). It leads to the "double decent" phenomenon that allows them to generalize in ways traditional statistical models can't. The idea that the "large" descriptor was just a subjective exclamation, like "oh wow this model is pretty large ain't it", is revisionism.
- danielmarkbruce 4mo agoyes, it does. That's why OpenAI refers to it's small models as small. They are just so different. The capabilities have changed dramatically. The use cases are wildly different. The architectures are quite different. Even the core idea of attention is different. Training them is materially different. Serving them is materially different. A 1.5 bill parameter model from 2019 is so different from today's LLMs that they really don't have much in common. What we have now is quite similar to what we had a couple years ago though.
- bbor 4mo agoThe term doesn't change its meaning because something new comes along. ...you're gonna flip when you hear about how language works :)
- Yizahi 4mo agoSure we do, since Fei-Fei Li and team created that annotated dataset, which allowed to train first LLMs. So LLMs are here for more than a decade already.
- danielmarkbruce 4mo agoYou are confused by what the L and L mean in LLM, or which data set she created, or both, or in general.
- Yizahi 4mo agoOr it is you who are confused. And I want to remind you that you can't retcon historical word use.
- danielmarkbruce 4mo agoFei Fei was annotating images... the second L in LLM is for "language". The first language models named LLM at the time were trained on language data, with an objective function of predicting the next token. It had nothing to do with the imagenet data. Imagenet data was used in... vision models. The attention is all you need paper didn't ever use the term LLM or large language model because the phrase didn't exist in industry. Why comment on a field you know nothing about?
- Nevermark 4mo agoI took humans thousands of years, then hundreds of years, to come to terms with very basic concepts about numbers. Its amazing to me when people talk about recombining things, or following up on things as somehow lesser work. People can't separate the perspective they were given when they learned the concepts, that those who developed the concepts didn't have because they didn't exist. Simple things are hard, or everything simple would have been done hundreds of years ago, and that is certainly not the case. Seeing something others have not noticed is very hard, when we don't have the concepts that the "invisible" things right in front of us will teach us.
- adi_kurian 4mo agoAnyone in the arts is aware that creativity is not the new, it is the repackaging of what already exists into something that is itself new.
- godelski 4mo agoIt's why the invention of teaching has been so important. Took a long time for humans to develop calculus. A long time to then refine it and make it much more useful. But then in a year or two an average person can learn what took hundreds of years to invent. It's crazy to equate these tasks as being the same. Even incremental innovation is difficult. You have to see something billions of people haven't. But there's also paradigm shifts and well... if you're not considered crazy at first then did you really shift a paradigm?
- nextaccountic 4mo agoFine, 8 years? That's not a long time
- asdfasgasdgasdg 4mo agoWhen people say this what they mean is that we've had plausibly useful LLMs for around three years, and I would say that is basically true.
- asdfasgasdgasdg 4mo agoWhen people say this what they mean is that we've had plausibly useful LLMs for around three years, and I would say that is basically true. The stuff before 2023 could barely be classified above the level of an interesting toy.