11 ms·
Meta has released the model weights for OPT-175B, which is used in the paper. There's also a lot of full release LLMs from other labs on the way as well.
by gaogao 4y ago
Meta has released the model weights for OPT-175B, which is used in the paper. There's also a lot of full release LLMs from other labs on the way as well.
- stavros 4y agoAre there any that perform anywhere close to GPT-3?
- jejeyyy77 4y agoStable Diffusion was behind DALLE, but has since surpassed it. Don't hear anyone talking about DALLE anymore.
- neilmovva 4y agoWhile OPT-175B is great to have publicly available, it needs a lot more training to achieve good results. Meta trained OPT on 180B tokens, compared to 300B that GPT-3 saw. And the Chinchilla scaling laws suggest that almost 4T tokens would be required to get the most bang for compute buck. And on top of that, there are some questions on the quality of open source data (The Pile) vs OpenAI’s proprietary dataset, which they seem to have spent a lot of effort cleaning. So: open source models are probably data-constrained, in both quantity and quality.
- mensetmanusman 4y agoIt’s fun to think about a few billion weights being the difference between useless and gold.
- tiborsaas 4y agoLooking at my bank account I can relate :)
- Miraste 4y agoOPT-175B isn't publicly available, sadly. It's available to research institutions, which is much better than "Open"AI, but it doesn't help us hobbyists/indie researchers much.
- generalizations 4y agoI wonder when we'll start putting these models on the pirate bay or similar. Seems like an excellent use for the tech. Has no one tried to upload OPT-175B anywhere like that yet?
- cma 4y agoIt could go on the clear net since trained weights aren't subject to copyright.