5 ms·
Training data has 0.339T tokens, less than the number of training parameters. A model like that could store all of the training text with 100B+ parameters left
by petters 5y ago
Training data has 0.339T tokens, less than the number of training parameters. A model like that could store all of the training text with 100B+ parameters left for computation.
- knuthsat 5y agoFor some reason this issue with model having insane amounts of weights but training data being small is not something that is an issue for modern NNs.
- lucidrains 5y agohttps://arxiv.org/abs/2109.02355 https://arxiv.org/abs/2109.02355
- thewarrior 5y agoMaybe that’s what it’s doing under the hood.
- visarga 5y agoBut then you try to predict the next token on a completely unseen piece of the corpus and fail miserably if all you do is store the training data.
- canjobear 5y agoParameters have been sufficient to memorize the training data for a while now. The fact that neural networks still generalize in this setting is a big mystery that is under active investigation.
- tgv 5y agoA single weight can’t encode an individual word, but the ratio looks close to overfitting too me too.
- petters 5y agoIf 16-bit floating-point numbers are used, it can presumably encode all tokens. In theory. It would not be very easy to work with.
- inciampati 5y agoI've often wondered if a lighter reinforcement learning based model on top of a full text index might do as well or better than these putatively overfit language models. Curious if anyone knows of ongoing or recent work on this approach.