2 ms·
I wonder how the number of parameters of this will compare to GPT-3 - will there be different sizes? Their stated goal is to make this available to academics/in
by sva_ 4y ago
I wonder how the number of parameters of this will compare to GPT-3 - will there be different sizes? Their stated goal is to make this available to academics/independent researchers/startups, and I think with a 175B parameter model, not having the trained model itself isn't necessarily the biggest hurdle for most people (having hardware with x hundred GB memory is - well maybe not for startups.)
Although there seems to be hope:
>In prior work, OpenAI found that the outputs from models trained with RLHF were preferred to those from 100x larger models trained without human feedback.
- deleted 4y ago[deleted]
- machiaweliczny 4y agoYeah, they say it will be "chinchilla-optimal", which means that it probably will be < 70B, might be actually much less as I've seen some recent work that 20B models are able to compete with GPT task [0] so I guess it might be using it but probably isn't, so there's further room for improvement. [0]https://www.reddit.com/r/MachineLearning/comments/y4tp4b/r_ul2_unifying_language_learning_paradigms_google/ https://www.reddit.com/r/MachineLearning/comments/y4tp4b/r_u...
- machiaweliczny 4y agoSeems like that 20B model is available on HuggingFace [0] [0] https://huggingface.co/docs/transformers/model_doc/ul2 https://huggingface.co/docs/transformers/model_doc/ul2
- wyldfire 4y agoWhat's "chinchilla"? is that a codename for a specific CPU/GPU? EDIT: no, it's a competitor with GPT-3 [1] [1] https://analyticsindiamag.com/deepmind-launches-gpt-3-rival-chinchilla/ https://analyticsindiamag.com/deepmind-launches-gpt-3-rival-...
- gradys 4y agoMore relevantly, it means that the model parameter count, dataset size, and training procedure are balanced such that they can achieve equal or better performance as GPT-3 while being smaller.
- jordn 4y agoThis is planned to be 70B but trained in the chinchilla-optimal way (more data + training). Scaling laws suggest this should outperform the base 175B GPT-3. Then release the base model as well as the RLHF-tuned models.