8 ms·
This is something that has repeatedly bothered me while playing with GPT-2. It retains too much "long-tail" knowledge that seems counterproductive for a generic
by colanderman 6y ago
This is something that has repeatedly bothered me while playing with GPT-2. It retains too much "long-tail" knowledge that seems counterproductive for a generic language tool. I would think those one-off associations should not be present after training has "digested" them but there they are.
- aabhay 6y agoHard to decide how much is too much.. the hard (and interesting) part of language is in the long tail
- alisonkisk 6y agoIt's over fitting, not long tail, that is the problem. It ends up copying input instead of analyzing and synthesizing.