3 ms·
I keep keep hearing people claim we are at the end of corpus scaling, but that seems totally unfounded. Where has it been proven you can’t running the training
by dougmwne 4y ago
I keep keep hearing people claim we are at the end of corpus scaling, but that seems totally unfounded. Where has it been proven you can’t running the training set through in multiple epochs in randomized order? Who’s to say you can’t collect all the non-English corpus and have the performance transfer to English? Who’s to say you can’t run the whole damn thing backwards and still have it learn something?
- zarzavat 4y agoWhen you train a NN, you train until the loss converges (or until you run out of money). If it happens that you exhaust your training data and the loss is still decreasing, then yes you keep going with a second epoch. On the other hand, if your loss has already converged then starting a new epoch on the same data is just burning money. In any case this wouldn’t be a data size scale but a model size scale. Model size scaling has its own set of limits, such as the aforementioned inference times. It seems safe to assume that OpenAI chose data and model size values for GPT-4 that were basically as big as they could go. We don’t know anything about GPT-4 or how it is trained or what the training data is, so from that perspective anything we can say about it is unfounded speculation. This is an important asterisk to any discussion of it.