3 ms·
There's a sufficient number of old texts - far more than necessary - if we were willing to converse with it in Latin*! Perhaps we could make a translator LLM to
by yyyk 3y ago
There's a sufficient number of old texts - far more than necessary - if we were willing to converse with it in Latin*! Perhaps we could make a translator LLM to converse with the 17th century LLM...
* I'm not talking about Roman-era Latin, Latin was the scholarly/international language for Europe for a long long time. AFAIK, most European Latin literature is untranslated...
- jstanley 3y agoAre you sure? In Andrej Karpathy's intro to LLMs[1], he says pre-training uses about 10TB of text from the web. It's hard to believe that a comparable amount of text had been created, even across all languages, in the entire history of humanity up to the 17th century. [1] https://www.youtube.com/watch?v=zjkBMFhNj_g https://www.youtube.com/watch?v=zjkBMFhNj_g
- yyyk 3y agoIf data quality is good, one can reach results with much less data: https://arxiv.org/abs/2305.07759 https://arxiv.org/abs/2305.07759
- thomasahle 3y agoHow many stories/tokens did they actually train on? I can't find it in the paper.
- yyyk 3y agoMaybe this will help? https://huggingface.co/datasets/roneneldan/TinyStories https://huggingface.co/datasets/roneneldan/TinyStories Also note https://arxiv.org/abs/2309.05463 https://arxiv.org/abs/2309.05463 (which is larger - obviously size still does contribute to performance)