3 ms·
> using the “show the model the data only once“ approach which is just wasteful. According to the InstructGPT paper, that is not the case, showing the data mul
by PartiallyTyped 3y ago
> using the “show the model the data only once“ approach which is just wasteful.
According to the InstructGPT paper, that is not the case, showing the data multiple times results in overfitting.
- whimsicalism 3y ago1. You are just referring to fine tuning, I am referring to training the base model. 2. They still saw performance improvements which is why they did train on the data multiple times, you can see in the paper. 3. there was a recent paper demonstrating that reusing data still saw continued improvements in perplexity, i am on my ipad so cannot find it now
- PartiallyTyped 3y ago> 1. You are just referring to fine tuning, I am referring to training the base model. Ahh mb! Sorry.