4 ms·
feels like large language models sucked all the air out of the room because it was a lot easier to scale compute and data, and after roberta, no one was willing
by htrp 2y ago
feels like large language models sucked all the air out of the room because it was a lot easier to scale compute and data, and after roberta, no one was willing to continue exploring.
- nshm 2y agoNo, there are mathematical reasons LLMs are better. They are trained with multiobjective loss (coding skills, translation skills, etc) so they understand the world much better than MLM. Original post discuss that but with more words and points than necessary.
- Der_Einzige 2y agoCall it a CLM vs MLM, not LLM vs MLM. Soon LMLM's will exist, which will be LLMs too...
- visarga 2y agoGPTs also get gradients from all tokens, BERT only on 15% masked tokens. GPTs are more effective.
- riku_iki 2y agoT5 is LLM, I think first one of them.