3 ms·
If data quality is good, one can reach results with much less data: https://arxiv.org/abs/2305.07759 https://arxiv.org/abs/2305.07759
by yyyk 3y ago
If data quality is good, one can reach results with much less data:
https://arxiv.org/abs/2305.07759 https://arxiv.org/abs/2305.07759
- thomasahle 3y agoHow many stories/tokens did they actually train on? I can't find it in the paper.
- yyyk 3y agoMaybe this will help? https://huggingface.co/datasets/roneneldan/TinyStories https://huggingface.co/datasets/roneneldan/TinyStories Also note https://arxiv.org/abs/2309.05463 https://arxiv.org/abs/2309.05463 (which is larger - obviously size still does contribute to performance)