3 ms·
> there are no regularizers in any modern LLMs. Using a large & diverse training set is the best regulariser, but I think there is also weight decay and dropou
by visarga 3y ago
> there are no regularizers in any modern LLMs.
Using a large & diverse training set is the best regulariser, but I think there is also weight decay and dropout in transformers
- euclaise 3y agoRWKV also uses some sort of L2-esque regularization, which was supposedly an idea taken from PaLM (although I can't find a source on this point, other than some message in the RWKV discord)