9 ms·

Optimized small model training is not only important for availability but also for the scientific study of LLMs. It’s like the use of simple organisms like yeas

Optimized small model training is not only important for availability but also for the scientific study of LLMs. It’s like the use of simple organisms like yeast for biological studies - we also need to study the simplest possible transformers that exhibit behaviors of interest from the larger models if we hope to ever understand LLMs and have more control over their behavior.

biophysboy 1y ago
It’s a fun analogy because the data “environment” of the model being trained matters a great deal
- jebarker 1y ago
  Exactly. YOLO runs of frontier models with a single random seed/data shuffle are pretty limited for trying to study the “molecular biology”. I actually like to think of LLM understanding as being like biology in the 1850s. There's lots of inspiration to be found in how biology has advanced since then and the types of experiments we might run to better understand LLMs.
  - biophysboy 1y ago
    Its something I keep thinking about when I see all these deep-dives by Anthropic on the "genetics" of LLMs. I see the emergent properties of LLMs as inseparable from their data environment. If the organization/prevalence of text online was different, I think Anthropic would see different "genetics". As the amt of LLM-generated text grows, I think it will become more clear that the "fundamental unit" is their relationship.
willvarfar 1y ago
(there are also lots of private company datasets like e.g. user purchase history that can be used with small models to solve real business problems. All the advances in 'large' language models can be leveraged and applied to small problems if the input sequences can be represented as a special custom language.)
smeeth 1y ago
I've been annoyed for a while people don't use a common parameter weight/compute budget for benchmarking papers. That said, it does make it easier to claim progress...
- pizza 1y ago
  https://github.com/KellerJordan/modded-nanogpt https://github.com/KellerJordan/modded-nanogpt is pretty great in that respect
- godelski 1y ago
  As a researcher, I can totally agree, but at the same time this isn't super straight forward. Things get weird because you can't just translate from one GPU to another. There isn't a clean calculation for that. There's also other issues like parallelism. Sure, your model is stable with a batch size of 8192 but that's across 1 node, it might not be stable with that batch across 2 nodes. This is a real frustrating part and honestly I don't think most people even are aware such issues exist. Right now I'm just happy when people are including parameter, GMACs (or FLOPs), and throughput. I always include those and the GPUs I used. I also frequently include more information in the appendix but frankly when I include it in the front matter the paper is more likely to be rejected. I can tell you why this isn't happening though. There's a common belief that scale is all you need. Which turns into "fuck the GPU poor". I've published works where my model is 100x smaller (with higher throughput, and far lower training costs), and the responses from reviewers tend to be along the lines "why isn't it better?" or "why not just distill or prune a large model?" There's this weird behavior that makes the black box stay a black box. I mean Yi Tay famously said "Fuck theorists" on twitter
ai-christianson 1y ago
I'm interested in one that can run fast on a laptop, but training can take a few days (maybe even longer) on the same laptop.
arethuza 1y ago
Thanks - that's one of the most interesting comments I've seen about LLMs. Makes me want to try training a model to sing "Daisy, Daisy..."
azath92 1y ago
Totally agree, one of the most interesting podcasts i have listened to in a while was a couple of years ago on the Tiny Stories paper and dataset (the author used that dataset) which focuses on stories that only contain simple words and concepts (like bedtime stories for a 3 year old), but which can be used to train smaller models to produce coherent english, both with grammar, diversity, and reasoning. The podcast itself with one of the authors was fantastic for explaining and discussing the capabilities of LLMs more broadly, using this small controlled research example. As an aside: i dont know what the dataset is in the biological analogy, maybe the agar plate. A super simple and controlled environment in which to study simple organisms. For ref: - Podcast ep https://www.cognitiverevolution.ai/the-tiny-model-revolution-with-ronen-eldan-and-yuanzhi-li-of-microsoft-research/ https://www.cognitiverevolution.ai/the-tiny-model-revolution... - tinystories paper https://arxiv.org/abs/2305.07759 https://arxiv.org/abs/2305.07759
- momojo 1y ago
  I like the agar plate analogy. Of course, the yeast is the star of the show, but so much work goes into prepping the plate. As someone in biotech, 90% of the complaints I hear over lunch are not about bad results, but about bad mistakes during the experiment. E.G. someone didn't cover their mouth while pipetting and the plates unusable now.
- re5i5tor 1y ago
  Ha! I remember where I was when I listened to that episode (Lakeshore Drive almost into Chicago for some event or other) — thanks for triggering that memory — super interesting stuff
leopoldj 1y ago
What the author is doing here is pre-training. This is something usually model makers like Google and Meta need to do. Most business are much better off doing fine-tuning or to a lesser extent continued pre-training. The author is doing this for academic reasons.
tmule 1y ago
Unfortunately, as things stand, it’s well-known that behaviors and optimizations in small scale models fail to replicate in larger models.
- jebarker 1y ago
  Well-known but not well-understood
- victorbjorklund 1y ago
  Which in itself is very interesting and requires study.
  - anvuong 1y ago
    It mostly has to do with sparsity in high dimensional space. When you scale things to the extreme everything is very far away from each other, the space is sparse, and random vectors have very high chance to be orthogonal, etc. All of these makes optimization incredibly slow and difficult. Just another facet of the so called "curse of dimensionality".
- indoordin0saur 1y ago
  But why? If we don't know why then how do we figure it out?
- yorwba 1y ago
  Doing hyperparameter sweeps on lots of small models to find the optimal values for each size and fitting scaling laws to predict the hyperparameters to use for larger models seems to work reasonably well. I think https://arxiv.org/abs/2505.01618 https://arxiv.org/abs/2505.01618 is the latest advance in that vein.
  - swyx 1y ago
    the problem is that the eval processes dont really work here if you believe in "Emergent Abilities" https://arxiv.org/abs/2206.07682 https://arxiv.org/abs/2206.07682
moojacob 1y ago
Enough with big data! Who's working on small data? https://www.youtube.com/watch?v=eDr6_cMtfdA&pp=ygUKc21hbGwgZGF0YQ%3D%3D https://www.youtube.com/watch?v=eDr6_cMtfdA&pp=ygUKc21hbGwgZ...

biophysboy 1y ago

It’s a fun analogy because the data “environment” of the model being trained matters a great deal

jebarker 1y ago
Exactly. YOLO runs of frontier models with a single random seed/data shuffle are pretty limited for trying to study the “molecular biology”. I actually like to think of LLM understanding as being like biology in the 1850s. There's lots of inspiration to be found in how biology has advanced since then and the types of experiments we might run to better understand LLMs.
- biophysboy 1y ago
  Its something I keep thinking about when I see all these deep-dives by Anthropic on the "genetics" of LLMs. I see the emergent properties of LLMs as inseparable from their data environment. If the organization/prevalence of text online was different, I think Anthropic would see different "genetics". As the amt of LLM-generated text grows, I think it will become more clear that the "fundamental unit" is their relationship.

willvarfar 1y ago

(there are also lots of private company datasets like e.g. user purchase history that can be used with small models to solve real business problems. All the advances in 'large' language models can be leveraged and applied to small problems if the input sequences can be represented as a special custom language.)

smeeth 1y ago

I've been annoyed for a while people don't use a common parameter weight/compute budget for benchmarking papers. That said, it does make it easier to claim progress...

pizza 1y ago
https://github.com/KellerJordan/modded-nanogpt https://github.com/KellerJordan/modded-nanogpt is pretty great in that respect
godelski 1y ago
As a researcher, I can totally agree, but at the same time this isn't super straight forward. Things get weird because you can't just translate from one GPU to another. There isn't a clean calculation for that. There's also other issues like parallelism. Sure, your model is stable with a batch size of 8192 but that's across 1 node, it might not be stable with that batch across 2 nodes. This is a real frustrating part and honestly I don't think most people even are aware such issues exist. Right now I'm just happy when people are including parameter, GMACs (or FLOPs), and throughput. I always include those and the GPUs I used. I also frequently include more information in the appendix but frankly when I include it in the front matter the paper is more likely to be rejected. I can tell you why this isn't happening though. There's a common belief that scale is all you need. Which turns into "fuck the GPU poor". I've published works where my model is 100x smaller (with higher throughput, and far lower training costs), and the responses from reviewers tend to be along the lines "why isn't it better?" or "why not just distill or prune a large model?" There's this weird behavior that makes the black box stay a black box. I mean Yi Tay famously said "Fuck theorists" on twitter

ai-christianson 1y ago

I'm interested in one that can run fast on a laptop, but training can take a few days (maybe even longer) on the same laptop.

arethuza 1y ago

Thanks - that's one of the most interesting comments I've seen about LLMs. Makes me want to try training a model to sing "Daisy, Daisy..."

azath92 1y ago

Totally agree, one of the most interesting podcasts i have listened to in a while was a couple of years ago on the Tiny Stories paper and dataset (the author used that dataset) which focuses on stories that only contain simple words and concepts (like bedtime stories for a 3 year old), but which can be used to train smaller models to produce coherent english, both with grammar, diversity, and reasoning. The podcast itself with one of the authors was fantastic for explaining and discussing the capabilities of LLMs more broadly, using this small controlled research example. As an aside: i dont know what the dataset is in the biological analogy, maybe the agar plate. A super simple and controlled environment in which to study simple organisms. For ref: - Podcast ep https://www.cognitiverevolution.ai/the-tiny-model-revolution-with-ronen-eldan-and-yuanzhi-li-of-microsoft-research/ https://www.cognitiverevolution.ai/the-tiny-model-revolution... - tinystories paper https://arxiv.org/abs/2305.07759 https://arxiv.org/abs/2305.07759

momojo 1y ago
I like the agar plate analogy. Of course, the yeast is the star of the show, but so much work goes into prepping the plate. As someone in biotech, 90% of the complaints I hear over lunch are not about bad results, but about bad mistakes during the experiment. E.G. someone didn't cover their mouth while pipetting and the plates unusable now.
re5i5tor 1y ago
Ha! I remember where I was when I listened to that episode (Lakeshore Drive almost into Chicago for some event or other) — thanks for triggering that memory — super interesting stuff

leopoldj 1y ago

What the author is doing here is pre-training. This is something usually model makers like Google and Meta need to do. Most business are much better off doing fine-tuning or to a lesser extent continued pre-training. The author is doing this for academic reasons.

tmule 1y ago

Unfortunately, as things stand, it’s well-known that behaviors and optimizations in small scale models fail to replicate in larger models.

jebarker 1y ago
Well-known but not well-understood
victorbjorklund 1y ago
Which in itself is very interesting and requires study.
- anvuong 1y ago
  It mostly has to do with sparsity in high dimensional space. When you scale things to the extreme everything is very far away from each other, the space is sparse, and random vectors have very high chance to be orthogonal, etc. All of these makes optimization incredibly slow and difficult. Just another facet of the so called "curse of dimensionality".
indoordin0saur 1y ago
But why? If we don't know why then how do we figure it out?
yorwba 1y ago
Doing hyperparameter sweeps on lots of small models to find the optimal values for each size and fitting scaling laws to predict the hyperparameters to use for larger models seems to work reasonably well. I think https://arxiv.org/abs/2505.01618 https://arxiv.org/abs/2505.01618 is the latest advance in that vein.
- swyx 1y ago
  the problem is that the eval processes dont really work here if you believe in "Emergent Abilities" https://arxiv.org/abs/2206.07682 https://arxiv.org/abs/2206.07682

moojacob 1y ago

Enough with big data! Who's working on small data? https://www.youtube.com/watch?v=eDr6_cMtfdA&pp=ygUKc21hbGwgZGF0YQ%3D%3D https://www.youtube.com/watch?v=eDr6_cMtfdA&pp=ygUKc21hbGwgZ...