4 ms·

I'm a big believer that smaller models will take hold. The throwing paint at the wall strategy of having 100B+ models and random data won't scale. It's possible

I'm a big believer that smaller models will take hold. The throwing paint at the wall strategy of having 100B+ models and random data won't scale. It's possible it will take new model architectures to get there. But having more fine-tuned control over model parameters is a pattern that will emerge. Then smaller, domain specific models can be joined together to form larger models, if full generalization is needed. I have an article on building micromodels that discusses some of this: https://neuml.hashnode.dev/train-a-language-model-from-scratch https://neuml.hashnode.dev/train-a-language-model-from-scrat...

pama 3y ago
Empirically, perhaps due to the lottery ticket hypothesis, one needs to wait a lot longer for small models to reach a useful loss, so if one is impatient it almost always pays off to train the largest model that a given compute configuration can support.
benrutter 3y ago
100%, I'd be amazed if we don't hit a time when the huge models we currently have are looked at the same way we see old fashioned room size computers now.
- bavell 3y ago
  Very apt analogy, stealing this :)
- whimsicalism 3y ago
  I would take the other side of that bet for sure. Modern computers don’t have less ram than the old fashioned room size computers. Amount of RAM seems much more analogous here than the size of the hardware running the software. Of course the latter will go down, but the former? I don’t see why.
  - tensor 3y ago
    Techniques like distilling show that you can in fact get similar performance with 60% fewer parameters. Yes it depends on making the large model first, but that only implies that somehow the large model is perhaps more useful in initial learning. And if that's the case, then it would be rather surprising if we couldn't identify why and create new training techniques that go directly to the smaller equivalent model. So in this way you've already lost your bet, as we already know that it's possible to create vastly smaller models with similar performance.
    - whimsicalism 3y ago
      60% is not vastly smaller in the way that I meant - we are talking about differences of three orders of magnitude in this article and that is what I am skeptical of. Nevertheless, I have not actually seen a model exhibiting high-level LM capabilities (of the GPT-4 level) that was distilled. Just because a model can match in perplexity with fewer parameters is not the same as matching the performance as subjectively measured by humanity interacting with the model.
      - klyrs 3y ago
        Early days, yet. If we could cut the size by 60% every few years, we'd be looking at a situation akin to Moore's law where a performant AI might fit in my grandkid's pocket.
    - fnordpiglet 3y ago
      That doesn’t mean that a much larger with high quality training data and new training techniques won’t be more powerful than a smaller model. We have incredible efficiencies in modern computers in addition to immense resources. They both are needed for the best performance. I suspect it’s the same for LLM. Superior data, superior techniques, and superior scale will yield superior models.
  - yayitswei 3y ago
    Information density will increase in models just as spatial density increased for hardware.
    - whimsicalism 3y ago
      Saying that information density will increase is very different from saying model sizes will decrease of a similar order of magnitude. Regardless, this isn't the trend we've seen with software RAM usage - information density is only decreasing as hardware becomes more abundant and comparatively less engineering effort goes into fitting everything into as small a portion as possible.
- dekhn 3y ago
  The computers I use are basically warehouse-sized now.
CuriouslyC 3y ago
You could always start with a big model then periodically prune low activation parameters as part of fine tuning until the desired size is reached.
api 3y ago
I’ve been wondering about variations on panel of experts designs that use a “mob” of small models or possibly a network of them. Sort of like crowdsourcing with baby AIs. Could we have libraries of small models that are all experts on niche subjects that basically collaborate? The advantage is that it’s easier to scale and distribute lots of small models on commodity hardware than it is to run giant models that require heaps of contiguous ram. I’d love to have some time to play with versions of this.
- nickpsecurity 3y ago
  The best design we know is how God made humans. Our brains have specialized parts that work together with analog components. We’re raised on knowledge of increasing complexity as humans, given general-purpose heuristics, and then we specialize. From there, small teams of specialists interacting typically do the best jobs. So, I’d try to do stuff like that first since it’s proven in the only general intelligence that exists. If I do experiments, I’m making models specialized from foundational to programming to specific language to specific tasks. Each pass will have training data so it picks up general concepts. Then, it will be trained to focus more on what it needs. Eventually, what’s slushing around in its neurons should be just the right information. Or it won’t work. Anyway, my concepts that I published are at this link: https://gethisword.com/tech/exploringai/index.html https://gethisword.com/tech/exploringai/index.html (See alternative models section.)
atorodius 3y ago
Agreed with the overall point (small models will prove powerful) but I disagree with: > But having more fine-tuned control over model parameters is a pattern that will emerge This assumes parameters do something interpretable. IMO, it’s more that parameters themselves are meaningless/dumb and patterns evolve via interaction of many such units (like in ants)
gwern 3y ago
> I'm a big believer that smaller models will take hold. The throwing paint at the wall strategy of having 100B+ models and random data won't scale. But all these smaller models are based on those 100b+ models! Like OP on TinyStories relies on larger models twice, to generate and then evaluate. Or the recent wave of small models which are all based on extracting data from GPT-3/4 to borrow the capbilities+RLHFing for free. You can talk about how interesting it is and how much of an overhang neural nets have in terms of being overparameterized (which is an important AI safety/capabilities issue - the first AGI will be the largest, slowest, and worst one, by a long shot), but the one thing it doesn't tell you is that smaller models can replace big models entirely. Because they are parasitic on said big models, so you still need to train the big models to begin with. (The smaller models also seem to lose a lot of the qualitative capabilities of larger ones, like meta-learning. Their TinyStories model does only stories. I doubt you could get it to 'dax a blick'.)
- benxh 3y ago
  Not all of them per se, take a look at something like Mistral. It's a 7B model displaying incredible performance. IMO, we still haven't even scratched the surface of what is possible with small LLMs. Especially not with pre-filtered/classified pre-training data. (Interesting LLMs based on their data approach and relatively small size: Qwen, InternLM, Mistral, Phi)
  - pama 3y ago
    In the context of this article, the small/tiny models were 1--30 million parameters and the large model was 1.5 billion parameters. Efficient training of a 7-billion parameter model already requires algorithms for training across multiple GPU because the memory requirement for derivatives and optimizers will not fit on the typical 80G Ram of the current high end GPUs.
  - brrrrrm 3y ago
    mind boggling that 7B is now considered a small model. I think it's valid, given the preeminence of 70B+ sized models. But wow, the community really just leap frogged over single digit billion parameter sizes.
whimsicalism 3y ago
I disagree - larger models are significantly more capable, likely as a function of their size, and the cost to inference them is only going to go down. Maybe for niche use cases like handling customer support requests small models will do well, but GPT4 is not going to be condensed to a few billion anytime soon imo.
logicchains 3y ago
There are hard limits to what a "brain" of a given size can do. Expecting ensembles of 7B models to match what properly trained 180B model can do at specialised tasks is like expecting a thousand monkeys trained on a specialised task to be able to match human performance.
- samatman 3y ago
  If this were true, then GPT4 would be better at chess than Stockfish. Since it's not, it isn't.
  - Terr_ 3y ago
    What if there was a large inefficient model just good enough that it could generate small specialized ones and orchestrate them? Now that I've said it, I wonder if there's any evidence human brains work that way.
  - taneq 3y ago
    By that logic, humans should be better at chess than Stockfish, but they aren’t.
zooch 3y ago
I would like to have a more modular approach with more specialized training of the models. Currently I can only really use smaller models for micro-tasks like sentiment analysis or classification, but any type of problem solving has to be left to GPT-4.
blueboo 3y ago
Believing this is existential for a thousand AI seed-stage startups. I hope it’s true for their sakes. Hear the other timeline from Darius Amodei. https://a16z.com/improving-ai/ https://a16z.com/improving-ai/ We may be entering a regime of multi-billion training costs on custom hardware. Open-source AI will be like open-source CPUs. Cool but the real thing even at the smallest scale comes out of monopsonized (is that a word?) inaccessible infrastructure
lyapunova 3y ago
I appreciate the conciseness of this article. Thank you for sharing. I agree. The kitchen sink data strategy appears to be fairly inefficient with current model architectures.

pama 3y ago

Empirically, perhaps due to the lottery ticket hypothesis, one needs to wait a lot longer for small models to reach a useful loss, so if one is impatient it almost always pays off to train the largest model that a given compute configuration can support.

benrutter 3y ago

100%, I'd be amazed if we don't hit a time when the huge models we currently have are looked at the same way we see old fashioned room size computers now.

bavell 3y ago
Very apt analogy, stealing this :)
whimsicalism 3y ago
I would take the other side of that bet for sure. Modern computers don’t have less ram than the old fashioned room size computers. Amount of RAM seems much more analogous here than the size of the hardware running the software. Of course the latter will go down, but the former? I don’t see why.
- tensor 3y ago
  Techniques like distilling show that you can in fact get similar performance with 60% fewer parameters. Yes it depends on making the large model first, but that only implies that somehow the large model is perhaps more useful in initial learning. And if that's the case, then it would be rather surprising if we couldn't identify why and create new training techniques that go directly to the smaller equivalent model. So in this way you've already lost your bet, as we already know that it's possible to create vastly smaller models with similar performance.
  - whimsicalism 3y ago
    60% is not vastly smaller in the way that I meant - we are talking about differences of three orders of magnitude in this article and that is what I am skeptical of. Nevertheless, I have not actually seen a model exhibiting high-level LM capabilities (of the GPT-4 level) that was distilled. Just because a model can match in perplexity with fewer parameters is not the same as matching the performance as subjectively measured by humanity interacting with the model.
    - klyrs 3y ago
      Early days, yet. If we could cut the size by 60% every few years, we'd be looking at a situation akin to Moore's law where a performant AI might fit in my grandkid's pocket.
  - fnordpiglet 3y ago
    That doesn’t mean that a much larger with high quality training data and new training techniques won’t be more powerful than a smaller model. We have incredible efficiencies in modern computers in addition to immense resources. They both are needed for the best performance. I suspect it’s the same for LLM. Superior data, superior techniques, and superior scale will yield superior models.
- yayitswei 3y ago
  Information density will increase in models just as spatial density increased for hardware.
  - whimsicalism 3y ago
    Saying that information density will increase is very different from saying model sizes will decrease of a similar order of magnitude. Regardless, this isn't the trend we've seen with software RAM usage - information density is only decreasing as hardware becomes more abundant and comparatively less engineering effort goes into fitting everything into as small a portion as possible.
dekhn 3y ago
The computers I use are basically warehouse-sized now.

CuriouslyC 3y ago

You could always start with a big model then periodically prune low activation parameters as part of fine tuning until the desired size is reached.

api 3y ago

I’ve been wondering about variations on panel of experts designs that use a “mob” of small models or possibly a network of them. Sort of like crowdsourcing with baby AIs. Could we have libraries of small models that are all experts on niche subjects that basically collaborate? The advantage is that it’s easier to scale and distribute lots of small models on commodity hardware than it is to run giant models that require heaps of contiguous ram. I’d love to have some time to play with versions of this.

nickpsecurity 3y ago
The best design we know is how God made humans. Our brains have specialized parts that work together with analog components. We’re raised on knowledge of increasing complexity as humans, given general-purpose heuristics, and then we specialize. From there, small teams of specialists interacting typically do the best jobs. So, I’d try to do stuff like that first since it’s proven in the only general intelligence that exists. If I do experiments, I’m making models specialized from foundational to programming to specific language to specific tasks. Each pass will have training data so it picks up general concepts. Then, it will be trained to focus more on what it needs. Eventually, what’s slushing around in its neurons should be just the right information. Or it won’t work. Anyway, my concepts that I published are at this link: https://gethisword.com/tech/exploringai/index.html https://gethisword.com/tech/exploringai/index.html (See alternative models section.)

atorodius 3y ago

Agreed with the overall point (small models will prove powerful) but I disagree with: > But having more fine-tuned control over model parameters is a pattern that will emerge This assumes parameters do something interpretable. IMO, it’s more that parameters themselves are meaningless/dumb and patterns evolve via interaction of many such units (like in ants)

gwern 3y ago

> I'm a big believer that smaller models will take hold. The throwing paint at the wall strategy of having 100B+ models and random data won't scale. But all these smaller models are based on those 100b+ models! Like OP on TinyStories relies on larger models twice, to generate and then evaluate. Or the recent wave of small models which are all based on extracting data from GPT-3/4 to borrow the capbilities+RLHFing for free. You can talk about how interesting it is and how much of an overhang neural nets have in terms of being overparameterized (which is an important AI safety/capabilities issue - the first AGI will be the largest, slowest, and worst one, by a long shot), but the one thing it doesn't tell you is that smaller models can replace big models entirely. Because they are parasitic on said big models, so you still need to train the big models to begin with. (The smaller models also seem to lose a lot of the qualitative capabilities of larger ones, like meta-learning. Their TinyStories model does only stories. I doubt you could get it to 'dax a blick'.)

benxh 3y ago
Not all of them per se, take a look at something like Mistral. It's a 7B model displaying incredible performance. IMO, we still haven't even scratched the surface of what is possible with small LLMs. Especially not with pre-filtered/classified pre-training data. (Interesting LLMs based on their data approach and relatively small size: Qwen, InternLM, Mistral, Phi)
- pama 3y ago
  In the context of this article, the small/tiny models were 1--30 million parameters and the large model was 1.5 billion parameters. Efficient training of a 7-billion parameter model already requires algorithms for training across multiple GPU because the memory requirement for derivatives and optimizers will not fit on the typical 80G Ram of the current high end GPUs.
- brrrrrm 3y ago
  mind boggling that 7B is now considered a small model. I think it's valid, given the preeminence of 70B+ sized models. But wow, the community really just leap frogged over single digit billion parameter sizes.

whimsicalism 3y ago

I disagree - larger models are significantly more capable, likely as a function of their size, and the cost to inference them is only going to go down. Maybe for niche use cases like handling customer support requests small models will do well, but GPT4 is not going to be condensed to a few billion anytime soon imo.

logicchains 3y ago

There are hard limits to what a "brain" of a given size can do. Expecting ensembles of 7B models to match what properly trained 180B model can do at specialised tasks is like expecting a thousand monkeys trained on a specialised task to be able to match human performance.

samatman 3y ago
If this were true, then GPT4 would be better at chess than Stockfish. Since it's not, it isn't.
- Terr_ 3y ago
  What if there was a large inefficient model just good enough that it could generate small specialized ones and orchestrate them? Now that I've said it, I wonder if there's any evidence human brains work that way.
- taneq 3y ago
  By that logic, humans should be better at chess than Stockfish, but they aren’t.

zooch 3y ago

I would like to have a more modular approach with more specialized training of the models. Currently I can only really use smaller models for micro-tasks like sentiment analysis or classification, but any type of problem solving has to be left to GPT-4.

blueboo 3y ago

Believing this is existential for a thousand AI seed-stage startups. I hope it’s true for their sakes. Hear the other timeline from Darius Amodei. https://a16z.com/improving-ai/ https://a16z.com/improving-ai/ We may be entering a regime of multi-billion training costs on custom hardware. Open-source AI will be like open-source CPUs. Cool but the real thing even at the smallest scale comes out of monopsonized (is that a word?) inaccessible infrastructure

lyapunova 3y ago

I appreciate the conciseness of this article. Thank you for sharing. I agree. The kitchen sink data strategy appears to be fairly inefficient with current model architectures.