5 ms·

100%, I'd be amazed if we don't hit a time when the huge models we currently have are looked at the same way we see old fashioned room size computers now.

bavell 3y ago
Very apt analogy, stealing this :)
whimsicalism 3y ago
I would take the other side of that bet for sure. Modern computers don’t have less ram than the old fashioned room size computers. Amount of RAM seems much more analogous here than the size of the hardware running the software. Of course the latter will go down, but the former? I don’t see why.
- tensor 3y ago
  Techniques like distilling show that you can in fact get similar performance with 60% fewer parameters. Yes it depends on making the large model first, but that only implies that somehow the large model is perhaps more useful in initial learning. And if that's the case, then it would be rather surprising if we couldn't identify why and create new training techniques that go directly to the smaller equivalent model. So in this way you've already lost your bet, as we already know that it's possible to create vastly smaller models with similar performance.
  - whimsicalism 3y ago
    60% is not vastly smaller in the way that I meant - we are talking about differences of three orders of magnitude in this article and that is what I am skeptical of. Nevertheless, I have not actually seen a model exhibiting high-level LM capabilities (of the GPT-4 level) that was distilled. Just because a model can match in perplexity with fewer parameters is not the same as matching the performance as subjectively measured by humanity interacting with the model.
    - klyrs 3y ago
      Early days, yet. If we could cut the size by 60% every few years, we'd be looking at a situation akin to Moore's law where a performant AI might fit in my grandkid's pocket.
  - fnordpiglet 3y ago
    That doesn’t mean that a much larger with high quality training data and new training techniques won’t be more powerful than a smaller model. We have incredible efficiencies in modern computers in addition to immense resources. They both are needed for the best performance. I suspect it’s the same for LLM. Superior data, superior techniques, and superior scale will yield superior models.
- yayitswei 3y ago
  Information density will increase in models just as spatial density increased for hardware.
  - whimsicalism 3y ago
    Saying that information density will increase is very different from saying model sizes will decrease of a similar order of magnitude. Regardless, this isn't the trend we've seen with software RAM usage - information density is only decreasing as hardware becomes more abundant and comparatively less engineering effort goes into fitting everything into as small a portion as possible.
dekhn 3y ago
The computers I use are basically warehouse-sized now.

bavell 3y ago

Very apt analogy, stealing this :)

whimsicalism 3y ago

I would take the other side of that bet for sure. Modern computers don’t have less ram than the old fashioned room size computers. Amount of RAM seems much more analogous here than the size of the hardware running the software. Of course the latter will go down, but the former? I don’t see why.

tensor 3y ago
Techniques like distilling show that you can in fact get similar performance with 60% fewer parameters. Yes it depends on making the large model first, but that only implies that somehow the large model is perhaps more useful in initial learning. And if that's the case, then it would be rather surprising if we couldn't identify why and create new training techniques that go directly to the smaller equivalent model. So in this way you've already lost your bet, as we already know that it's possible to create vastly smaller models with similar performance.
- whimsicalism 3y ago
  60% is not vastly smaller in the way that I meant - we are talking about differences of three orders of magnitude in this article and that is what I am skeptical of. Nevertheless, I have not actually seen a model exhibiting high-level LM capabilities (of the GPT-4 level) that was distilled. Just because a model can match in perplexity with fewer parameters is not the same as matching the performance as subjectively measured by humanity interacting with the model.
  - klyrs 3y ago
    Early days, yet. If we could cut the size by 60% every few years, we'd be looking at a situation akin to Moore's law where a performant AI might fit in my grandkid's pocket.
- fnordpiglet 3y ago
  That doesn’t mean that a much larger with high quality training data and new training techniques won’t be more powerful than a smaller model. We have incredible efficiencies in modern computers in addition to immense resources. They both are needed for the best performance. I suspect it’s the same for LLM. Superior data, superior techniques, and superior scale will yield superior models.
yayitswei 3y ago
Information density will increase in models just as spatial density increased for hardware.
- whimsicalism 3y ago
  Saying that information density will increase is very different from saying model sizes will decrease of a similar order of magnitude. Regardless, this isn't the trend we've seen with software RAM usage - information density is only decreasing as hardware becomes more abundant and comparatively less engineering effort goes into fitting everything into as small a portion as possible.

dekhn 3y ago

The computers I use are basically warehouse-sized now.