4 ms·
I could be wrong, but I thought the llama 2 paper explicitly called out 1 epoch and that more than that caused over-fitting in their other experiments.
by danielmarkbruce 3y ago
I could be wrong, but I thought the llama 2 paper explicitly called out 1 epoch and that more than that caused over-fitting in their other experiments.
- whimsicalism 3y agoYou're not at all wrong :) I think a lot of people confuse the pre-training and fine-tuning runs because these are all novel concepts.
- computerex 3y agoLlama: https://arxiv.org/pdf/2302.13971.pdf https://arxiv.org/pdf/2302.13971.pdf You can clearly see in table on second page that higher quality data is trained on more than 1 epoch. Most Open LLM's do this.
- whimsicalism 3y agoLlama2 doesn’t and outclasses llama. I believe GPT was trained in same manner
- computerex 3y ago> I believe GPT was trained in same manner If you are talking about GPT-4, unless you are insider (doubt it) you'd have no way of proving it either way because that info is not public. > Llama2 doesn’t and outclasses llama. My point still stands, llama2 is just one llm, and we still don't know the distribution of their training set.