3 ms·

Probably not enough old texts for it to learn how the language and the world work. Hence the fine tuning.

aroo 3y ago
Sounds like something right up the domain of synthetic data.
yyyk 3y ago
There's a sufficient number of old texts - far more than necessary - if we were willing to converse with it in Latin*! Perhaps we could make a translator LLM to converse with the 17th century LLM... * I'm not talking about Roman-era Latin, Latin was the scholarly/international language for Europe for a long long time. AFAIK, most European Latin literature is untranslated...
- jstanley 3y ago
  Are you sure? In Andrej Karpathy's intro to LLMs[1], he says pre-training uses about 10TB of text from the web. It's hard to believe that a comparable amount of text had been created, even across all languages, in the entire history of humanity up to the 17th century. [1] https://www.youtube.com/watch?v=zjkBMFhNj_g https://www.youtube.com/watch?v=zjkBMFhNj_g
  - yyyk 3y ago
    If data quality is good, one can reach results with much less data: https://arxiv.org/abs/2305.07759 https://arxiv.org/abs/2305.07759
    - thomasahle 3y ago
      How many stories/tokens did they actually train on? I can't find it in the paper.
      - yyyk 3y ago
        Maybe this will help? https://huggingface.co/datasets/roneneldan/TinyStories https://huggingface.co/datasets/roneneldan/TinyStories Also note https://arxiv.org/abs/2309.05463 https://arxiv.org/abs/2309.05463 (which is larger - obviously size still does contribute to performance)
Dorialexander 3y ago
Yes. I think we may have enough for "full finetuning" and erasing to a large extent the previous knowledge. But that's still very far off for pretraining. "RomeGPT" is next on my list of Monad successors and to give you a general idea, we have on the order of tens of millions of words in classical Latin (and biggest source will… Augustine). There was a BERT Latin project that was able to collect roughly 500 million words in all with mostly early modern and modern Latin. In comparison I'm currently part of a project to pretrain a French model and we need… 140 billion words.

aroo 3y ago

Sounds like something right up the domain of synthetic data.

yyyk 3y ago

There's a sufficient number of old texts - far more than necessary - if we were willing to converse with it in Latin*! Perhaps we could make a translator LLM to converse with the 17th century LLM... * I'm not talking about Roman-era Latin, Latin was the scholarly/international language for Europe for a long long time. AFAIK, most European Latin literature is untranslated...

jstanley 3y ago
Are you sure? In Andrej Karpathy's intro to LLMs[1], he says pre-training uses about 10TB of text from the web. It's hard to believe that a comparable amount of text had been created, even across all languages, in the entire history of humanity up to the 17th century. [1] https://www.youtube.com/watch?v=zjkBMFhNj_g https://www.youtube.com/watch?v=zjkBMFhNj_g
- yyyk 3y ago
  If data quality is good, one can reach results with much less data: https://arxiv.org/abs/2305.07759 https://arxiv.org/abs/2305.07759
  - thomasahle 3y ago
    How many stories/tokens did they actually train on? I can't find it in the paper.
    - yyyk 3y ago
      Maybe this will help? https://huggingface.co/datasets/roneneldan/TinyStories https://huggingface.co/datasets/roneneldan/TinyStories Also note https://arxiv.org/abs/2309.05463 https://arxiv.org/abs/2309.05463 (which is larger - obviously size still does contribute to performance)

Dorialexander 3y ago

Yes. I think we may have enough for "full finetuning" and erasing to a large extent the previous knowledge. But that's still very far off for pretraining. "RomeGPT" is next on my list of Monad successors and to give you a general idea, we have on the order of tens of millions of words in classical Latin (and biggest source will… Augustine). There was a BERT Latin project that was able to collect roughly 500 million words in all with mostly early modern and modern Latin. In comparison I'm currently part of a project to pretrain a French model and we need… 140 billion words.