4 ms·
Can anyone provide details about the training of the model. What data is it based on? Common Crawl? (Being a french company the also rather focus on English lan
by riedel 3y ago
Can anyone provide details about the training of the model. What data is it based on? Common Crawl? (Being a french company the also rather focus on English language tasks) Where was it trained and with how many resources? They mention Leonardo. I was in an interesting meeting at the German Research Ministry last week where people where complaining that the EuroHPC resources were not sufficient atm to train decent LLMs. I guess the guys in the end went also to CoreWeave in the US.
- brucethemoose2 3y ago> Inspecting the tokenizer model, there is evidence indicated a training dataset of 8T tokens (/mnt/test/datasets/tokenizer_training/8T_train_data/shuffled.txt) https://github.com/ggerganov/llama.cpp/pull/3362#issuecomment-1737333406 https://github.com/ggerganov/llama.cpp/pull/3362#issuecommen...
- deleted 3y ago[deleted]
- brucethemoose2 3y ago(For anyone reading, this is apparently false, according to some Discord post)