4 ms·
Hasn't it already been trained on what is effectively the entire contents of the scrapable internet? There isn't another 10x to be had there, let alone 100x. I
by eslaught 4y ago
Hasn't it already been trained on what is effectively the entire contents of the scrapable internet? There isn't another 10x to be had there, let alone 100x.
I assume that whatever future improvements we get from improving algorithms (or perhaps through throwing more compute at it), not through larger datasets.
- eru 4y agoThere might not be another 100x of written language. But we noticed that training your neural networks on multiple tasks actually works well. So we could start feeding our models eg audio and video. With lots of webcams we can make arbitrary amounts of new video footage. That would also allow the language model to be grounded more in our 3d reality. (Granted, we only know as a general observation that training the same network for multiple tasks 'forces' that network to become better and abstract and generalise. Nobody has yet publicly demonstrated an application of that observation to training language models + video models.) Another avenue: at the moment those large language models only see each example once, if I remember right. We still have lots of techniques for augmenting training data (eg via noise and dropout etc), or even just presenting the same data multiple times without overfitting.