2 ms·
In general, most giant LLMs are extremely undertrained at this time. Consider that most of the gains in RoBerta vs bert were from just continuing to train.
by Der_Einzige 4y ago
In general, most giant LLMs are extremely undertrained at this time. Consider that most of the gains in RoBerta vs bert were from just continuing to train.
- stevenhuang 4y agoCases of undertraining can be observed whenever the output is repeating gibberish or loops. Happened a lot in GPT2 ai dungeon days
- leobg 4y agoSo can we continue training RoBERTa to get it to, say, GPT3 Ada level