5 ms·
No its the opposite: overfitting is the result of either having too many weights compared to the size of your dataset, or training for a long time while reusing
by SuchAnonMuchWow 2y ago
No its the opposite: overfitting is the result of either having too many weights compared to the size of your dataset, or training for a long time while reusing/transforming parts of your dataset to make it last longer.
Having a huge dataset compared to the size of your network will reduce overfitting.
- Version467 2y agoWe don't actually know how big the Dataset is, right? It could be the same dataset used for Llama 2, but trained for more Epochs.
- GaggiX 2y agoThe dataset is 7 times bigger than the dataset used for Llama 2 as reported by Meta.
- baobabKoodaa 2y agoHas Meta disclosed how much parts of the dataset were repeated? I've only seen the "number of tokens trained" number.