2 ms·
Source? Maybe several epochs on some very small subsets, but my strong impression was that it was 1 epoch in the pre-training run for pretty much all of the top
by whimsicalism 3y ago
Source? Maybe several epochs on some very small subsets, but my strong impression was that it was 1 epoch in the pre-training run for pretty much all of the top LLMs.
- fpgaminer 3y agoLlama off the top of my head: https://arxiv.org/pdf/2302.13971.pdf https://arxiv.org/pdf/2302.13971.pdf
- danielmarkbruce 3y agoGood one, I certainly managed to forget this and had "1" firmly etched in my brain.