3 ms·
Model size does not necessarily correlates to quality of results. "Chinchilla (70B) Greatly Outperforms GPT-3 (175B) and Gopher (280B)" - https://towardsdatasc
by belter 4y ago
Model size does not necessarily correlates to quality of results.
"Chinchilla (70B) Greatly Outperforms GPT-3 (175B) and Gopher (280B)" - https://towardsdatascience.com/a-new-ai-trend-chinchilla-70b-greatly-outperforms-gpt-3-175b-and-gopher-280b-408b9b4510 https://towardsdatascience.com/a-new-ai-trend-chinchilla-70b...
- Der_Einzige 4y agoI highly doubt this in practice on a large scale. Outside of the common phenomena of "most large NNs are under trained" and "less better data is sometimes better than more worse data", there are no other obvious mechanisms to explain why a smaller model with same or similar architecture would be better than a larger one. I claim instead that we are still hardly scratching the surface with how we evaluate NLP systems. Also, some fields have straight up trash evaluation schemes. Summarization and ROGUE scores are totally BS and I find the claim that they even correlate with high quality summaries suspect. I say this with publications in the that subfield, so I have personal experience with just how crummy many summarizes are.
- WithinReason 4y agothere are no other obvious mechanisms to explain why a smaller model with same or similar architecture would be better than a larger one. Overfitting?
- Der_Einzige 4y agoThe consensus seems to be that the majority of LMs are undertrained not overfitting though.
- espadrine 4y agoAn interesting outcome of the nanoGPT repo is this struggle to exactly match the Chinchilla findings[0], even after discussing it with the authors. A larger discussion is that the scaling laws achieve loss-optimal compute time, but the pre-training loss only improves predictions on the corpus, which contains texts written by people that were wrong or whose prose was lacking. In a real system, what you want to optimize for is accuracy, composability, inventiveness. [0]: https://github.com/karpathy/nanoGPT/blob/master/scaling_laws.ipynb https://github.com/karpathy/nanoGPT/blob/master/scaling_laws...