4 ms·
Yeah. They say they are doing a 10B release soon[1]. I suspect they have run into training issues since they are moving to a new repo[2] [1] https://twitter.c
by nl 6y ago
Yeah. They say they are doing a 10B release soon[1].
I suspect they have run into training issues since they are moving to a new repo[2]
[1] https://twitter.com/arankomatsuzaki/status/1373732646811967490 https://twitter.com/arankomatsuzaki/status/13737326468119674...
[2] https://github.com/EleutherAI/gpt-neox/ https://github.com/EleutherAI/gpt-neox/
- chillee 6y agoIt's more about hardware - these models were trained on TPUs, while GPT-NeoX is being trained on GPUs graciously provided by Coreweave.
- orra 6y agoAny idea what the required GPU time would cost (if not donated)? Is GPT-3 just a commodity soon?
- Voloskaya 6y ago~4M$ per full training give or take.
- teruakohatu 6y agoThe number thrown around for gpt-3 is $4.6 million, but I am not sure where that figure originates.
- minimaxir 6y agoIt was a number tossed around by a GPU hosting provider, based on their own costs: https://lambdalabs.com/blog/demystifying-gpt-3/ https://lambdalabs.com/blog/demystifying-gpt-3/ The reality is that GPT-3 was likely "free" to train on Azure, as Microsoft has provided a lot of resources to OpenAI.
- exikyut 6y agoIf this is true, I wonder what sort of social capital transactional exchange is going on instead.
- minimaxir 6y agoWith training improvements such as DeepSpeed, the GPU costs will likely be substantially lower than what was available at the time OpenAI trained GPT-3. Still not free, though. The hard part with GPT-3 is it's big enough to make it difficult to actually deploy.
- stellaathena 6y agoOur current estimate is that it requires between 2000 and 4000 V100 months.