3 ms·
The TinyStories paper[0] explores some of the potential of "small language models." I think you don't need as much data if you have really high quality data (re
by ronsor 3y ago
The TinyStories paper[0] explores some of the potential of "small language models." I think you don't need as much data if you have really high quality data (really high quality); however, having a lot of data seems to be able to compensate for widely varying quality.
[0] https://ar5iv.labs.arxiv.org/html/2305.07759 https://ar5iv.labs.arxiv.org/html/2305.07759
- giardini 3y agoThank you for the reference! But as long as there is low quality data might not that data "pollute" the process (whatever that may be)? i.e., "GIGO". Is "low quality data" possibly necessary for a properly-working LLM?! [Aside: why the downvote?]