To elaborate on my comment on the other thread, a workaround to overfitting is to train on so much distinct data that the model can't overfit. Newer large datas

To elaborate on my comment on the other thread, a workaround to overfitting is to train on so much distinct data that the model can't overfit. Newer large datasets optimize for diversity, as the AI industry is slowing coming to the realization that better data is much more important than large amount of bad data for training LLMs. The writeup of SlimPajama, a heavily-deduped dataset, is a good starting point: https://www.cerebras.net/blog/slimpajama-a-627b-token-cleaned-and-deduplicated-version-of-redpajama https://www.cerebras.net/blog/slimpajama-a-627b-token-cleane...