3 ms·
Modern LLMs showed that overfitting disappears if you add more and more parameters. "Double descent" is well documented, if not well understood.
by warkdarrior 7mo ago
Modern LLMs showed that overfitting disappears if you add more and more parameters. "Double descent" is well documented, if not well understood.
- runarberg 7mo ago> Modern LLMs showed that overfitting disappears if you add more and more parameters. I have not seen that. In fact this is the first time I hear this claim, and frankly it sounds ludicrous. I don‘t know how modern LLMs are dealing with overfitting but I would guess there is simply a content matching algorithm after the inference, and if there is a copyright match the program does something to alter or block the generation. That is, I suspect the overfitting prevention is algorithmic and not part of the model.