5 ms·
Yeah, they say it will be "chinchilla-optimal", which means that it probably will be < 70B, might be actually much less as I've seen some recent work that 20B m
by machiaweliczny 4y ago
Yeah, they say it will be "chinchilla-optimal", which means that it probably will be < 70B, might be actually much less as I've seen some recent work that 20B models are able to compete with GPT task [0] so I guess it might be using it but probably isn't, so there's further room for improvement.
[0]https://www.reddit.com/r/MachineLearning/comments/y4tp4b/r_ul2_unifying_language_learning_paradigms_google/ https://www.reddit.com/r/MachineLearning/comments/y4tp4b/r_u...
- machiaweliczny 4y agoSeems like that 20B model is available on HuggingFace [0] [0] https://huggingface.co/docs/transformers/model_doc/ul2 https://huggingface.co/docs/transformers/model_doc/ul2
- wyldfire 4y agoWhat's "chinchilla"? is that a codename for a specific CPU/GPU? EDIT: no, it's a competitor with GPT-3 [1] [1] https://analyticsindiamag.com/deepmind-launches-gpt-3-rival-chinchilla/ https://analyticsindiamag.com/deepmind-launches-gpt-3-rival-...
- gradys 4y agoMore relevantly, it means that the model parameter count, dataset size, and training procedure are balanced such that they can achieve equal or better performance as GPT-3 while being smaller.