3 ms·
From the article: Pre-trained on a trillion "tokens"... Doesn't 7B indicates that it was trained on 7 billion tokens? Or am I misunderstanding the nomenclatur
by starik36 4y ago
From the article: Pre-trained on a trillion "tokens"...
Doesn't 7B indicates that it was trained on 7 billion tokens? Or am I misunderstanding the nomenclature?
- instance 4y ago7B is the number of parameters of the model.
- superkuh 4y agoThe emerging consensus for larger LLM is you want to train them with at least 2-4x the tokens of the number of parameters (weights between neurons in the layers). A trillion (100x) surprises me.
- sebzim4500 4y agoThey probably put most of the effort into the 65B model, the 7B model was just trained so they could get an idea of the scaling behaviour. It makes sense to use the same amount of training steps, then.
- sitic 4y agoThe LLaMA paper contradicts this view: "[...] Although Hoffmann et al. (2022) recommends training a 10B model on 200B tokens, we find that the performance of a 7B model continues to improve even after 1T tokens." https://arxiv.org/pdf/2302.13971.pdf https://arxiv.org/pdf/2302.13971.pdf
- dragonwriter 4y ago> Doesn't 7B indicates that it was trained on 7 billion tokens? No, 7B means it has 7 billion parameters.
- starik36 4y agoAnd what does a parameter mean in this context?
- dragonwriter 4y agoA parameter is the connection weight between two neurons in the network.
- deleted 4y ago[deleted]