3 ms·
Wouldn't it result in overfitting?
by zeruh 3y ago
Wouldn't it result in overfitting?
- kevsim 3y agoI would also think it'd be an incredibly expensive way to train a model.
- ReptileMan 3y agoDepends. I wonder what is the minimum reasonable amount of different tokens needed to lift up the weights.
- Jensson 3y agoYou store the output from ChatGPT, you don't run it again every time you do a training step. Generating millions of examples to add to your own training wont cost much at all relatively.
- triyambakam 3y agoThe details on how exactly they may have used it to train their model is vague. I believe transfer learning or knowledge distillation are valid techniques based on the inference from other models.