3 ms·
> if you want improved performance, you still need more data Not true. See figure 2: https://arxiv.org/pdf/2203.15556.pdf#page=5 https://arxiv.org/pdf/2203.155
by peaslock 4y ago
> if you want improved performance, you still need more data
Not true. See figure 2: https://arxiv.org/pdf/2203.15556.pdf#page=5 https://arxiv.org/pdf/2203.15556.pdf#page=5
The loss decreases with greater model size at the same compute budget (i.e. stopping sooner regarding training data). Also some rehearsal/multi-epoch training improves the forgetting rate (thereby improving performance substantially), which hasn't been taken into account by Chinchilla et al. because they train <1 epoch.
https://arxiv.org/abs/2205.12393 https://arxiv.org/abs/2205.12393
- zone411 4y agoNo. It shows the opposite. All model sizes converged to a similar loss as the compute increased towards maximum. But larger models had larger loss for a given compute budget. Their text about Figure 3 confirms what I'm saying: "We find a clear valley in loss, meaning that for a given FLOP budget there is an optimal model to train"
- peaslock 4y agoYes, but the losses in Figure 3 increase because the larger models see fewer data to keep the FLOP budget constant, not because of overfitting. Large models do not overfit very much, so the loss of a larger model will still be better compared to a smaller model when you keep dataset size constant.