3 ms·

Very interesting. According to their X posts, this meme model "SmolLm" beats Meta's new 1B and 3B models across almost all metrics. I wonder how this is possib

Very interesting. According to their X posts, this meme model "SmolLm" beats Meta's new 1B and 3B models across almost all metrics. I wonder how this is possible given that Meta has been in this game for much longer and probably has much more data at their disposal as well.

stavros 2y ago
Usually, that's because they use a groundbreaking ML method called TTDS, or "training on the test dataset".
- exe34 2y ago
  meta or smol?
  - stavros 2y ago
    Smol
- jgalt212 2y ago
  I would hope Simon would not fall victim to such shenanigans, and has his own test dataset.
  - stavros 2y ago
    I'm not saying either did this, just that that's what most fine tunes tend to do.
  - simonw 2y ago
    My test dataset is mostly dumb prompts about pelicans. You'll note that I didn't quote their benchmarks in my own post at all, because I didn't want to boost them without feeling confident in what they were stating. I posted about this because my own very limited initial experiments passed a loose vibe check! I'm impressed any time a 1.7GB (or 130MB) model file appears to be able to do anything useful at all.
- jerpint 2y ago
  Do you have proof for this? Why accuse one team and not the other?
  - stavros 2y ago
    Which team did I accuse? I said "usually".
    - deleted 2y ago
      [deleted]
wokwokwok 2y ago
You can be reasonably confident that unless there’s been a significant breakthrough (there hasn’t) if a smaller model beats a larger model it’s either fine tuned for a specific purpose or trained on the test data somehow (ie. fine tuned to have good metrics). To be less snarky, they claim: > These models are built on a meticulously curated high-quality training corpus Ie. Good training data plus a small model beats a bigger model. …but I’m skeptical, when I read: > We observed that performance continues to improve with longer training, even beyond the Chinchilla optimal point. Therefore, we decided to train the 1.7B model on 1 trillion tokens and the 135M and 360M models on 600B tokens, as the performance gains after 400B tokens begin to slow on some benchmarks for these smaller models. So they’re evaluating their models against various benchmarks as they train them and picking the practice that gives the best benchmarks? I dunno. The claim is basically good data > more parameters, but it’s just an observation of “this happened to work for us” rather than something you can usefully take (as far as I can see) and apply to larger models. The claims they actaully make about performance are far more modest than people are making out. The 1.7B model performs better than any other 2B models in their evaluation. Seems nice. Not ground breaking. Not convinced it’s real rather than polluted training data personally.
moffkalast 2y ago
Meta doesn't train on their internal data, at least not for open models. It would be a real PR problem if someone started dumping real Facebook chats out of them. And this is from Huggingface themselves, arguably they have a lot of data as well.
thinker567 2y ago
They didn’t say they beat the 3B, their 1.7B outperforms llama3.2 1B thanks to 11T tokens of high quality data (Hugging Face are the ones behind FineWeb dataset that everyone uses now). Btw Qwen2.5-1B also surpasses llama3.2 1B by a large margin so beating it is even more impressive

stavros 2y ago

Usually, that's because they use a groundbreaking ML method called TTDS, or "training on the test dataset".

exe34 2y ago
meta or smol?
- stavros 2y ago
  Smol
jgalt212 2y ago
I would hope Simon would not fall victim to such shenanigans, and has his own test dataset.
- stavros 2y ago
  I'm not saying either did this, just that that's what most fine tunes tend to do.
- simonw 2y ago
  My test dataset is mostly dumb prompts about pelicans. You'll note that I didn't quote their benchmarks in my own post at all, because I didn't want to boost them without feeling confident in what they were stating. I posted about this because my own very limited initial experiments passed a loose vibe check! I'm impressed any time a 1.7GB (or 130MB) model file appears to be able to do anything useful at all.
jerpint 2y ago
Do you have proof for this? Why accuse one team and not the other?
- stavros 2y ago
  Which team did I accuse? I said "usually".
  - deleted 2y ago
    [deleted]

wokwokwok 2y ago

You can be reasonably confident that unless there’s been a significant breakthrough (there hasn’t) if a smaller model beats a larger model it’s either fine tuned for a specific purpose or trained on the test data somehow (ie. fine tuned to have good metrics). To be less snarky, they claim: > These models are built on a meticulously curated high-quality training corpus Ie. Good training data plus a small model beats a bigger model. …but I’m skeptical, when I read: > We observed that performance continues to improve with longer training, even beyond the Chinchilla optimal point. Therefore, we decided to train the 1.7B model on 1 trillion tokens and the 135M and 360M models on 600B tokens, as the performance gains after 400B tokens begin to slow on some benchmarks for these smaller models. So they’re evaluating their models against various benchmarks as they train them and picking the practice that gives the best benchmarks? I dunno. The claim is basically good data > more parameters, but it’s just an observation of “this happened to work for us” rather than something you can usefully take (as far as I can see) and apply to larger models. The claims they actaully make about performance are far more modest than people are making out. The 1.7B model performs better than any other 2B models in their evaluation. Seems nice. Not ground breaking. Not convinced it’s real rather than polluted training data personally.

moffkalast 2y ago

Meta doesn't train on their internal data, at least not for open models. It would be a real PR problem if someone started dumping real Facebook chats out of them. And this is from Huggingface themselves, arguably they have a lot of data as well.

thinker567 2y ago

They didn’t say they beat the 3B, their 1.7B outperforms llama3.2 1B thanks to 11T tokens of high quality data (Hugging Face are the ones behind FineWeb dataset that everyone uses now). Btw Qwen2.5-1B also surpasses llama3.2 1B by a large margin so beating it is even more impressive