5 ms·
What do you mean by "small players have no chance"? OpenAI was founded in 2015, it used to be a "small player" which just got things right and grew with it - we
by spi 4y ago
What do you mean by "small players have no chance"? OpenAI was founded in 2015, it used to be a "small player" which just got things right and grew with it - we're not talking of Google or Facebook investing a chunk of their billions cash. In Germany, AlephAlpha has built their own supercomputer and are training similar sized models. It's expensive for sure, but well in the possibilities of startups. In France researchers trained the similarly sized BLOOM model https://huggingface.co/bigscience/bloom https://huggingface.co/bigscience/bloom. They claim it cost between $2 and $4 millions.
Sure, a single researcher can't replicate this at their university, but even though OpenAI likes to publish it this way, we're not really talking about research here. Research was inventing the transformer architecture, this is just making it bigger by (very smart) engineering choices. It's something companies should do (and are doing), not researchers.
- awestroke 4y agoMicrosoft (using Azure DCs) built a supercomputer with 10,000 V100 GPUs exclusively for OpenAI. [0] It is estimated that it cost around $5M in compute time to train GPT-3. OpenAI has received billions in investment prior to launching GPT-3, including $1B from Microsoft in 2019. [0]: https://blogs.microsoft.com/ai/openai-azure-supercomputer/ https://blogs.microsoft.com/ai/openai-azure-supercomputer/
- nileshtrivedi 4y ago> we're not talking of Google or Facebook investing a chunk of their billions cash OpenAI had raised $1B from Microsoft in 2019 and used it to train a 175B param model. Now, they have raised $10B and are training GPT-4 with 1.5T params. GPUs are capital intensive and as long as there are returns to bigger models, that's exactly where things will go.
- andy_ppp 4y agoWill 1.5T parameters be possible to run in the public way GPT-3 is? I can’t wait to see what happens with this much learning!
- awestroke 4y agoI can't find any source on the 1.5T params number. I'd love to read more if you have any links to share. Thanks
- wut42 4y agoafaik, gpt-4 is mostly rumours so far, same thing for the 1.5T number. gpt-4 is suerly coming.
- wnkrshm 4y agoMaybe it will be called GPT-XP by then, with Microsoft owning half of it.
- taneq 4y agoGPT-10 will be evergreen and 'the last version of GPT'. And then three years later GPT-11 will be required to run the latest games.
- ben_w 4y agoIn my imagination, OpenAI does what Bungie did when MS bought them, and open-sources what used to be their crown jewels. That said, GPT-AlephOne only makes sense if there's a preceding GPT-∞.
- egorfine 4y agoThey have got to release GPT-3.11 For Workgroups first.
- awestroke 4y agoOr GPT-365
- int_19h 4y agoWhy not ask GPT itself what it wants to be called?
- 4y ago
- orbifold 4y agoI am actually still unclear how AlephAlpha pulled that off and who funds them, since they have a rather low profile team.
- hdjjhhvvhga 4y ago> we're not talking of Google or Facebook investing a chunk of their billions cash. On the contrary, in this thread we are are mainly talking about that.
- SilverBirch 4y agoOpenAI was founded in 2015 by a group of billionaires who pledged $1Bn of funding. That is hardly a small scrappy start up.