5 ms·
How does the outfit intend to fund the project? OpenAI spends millions on computing resources to train the models.
by joshlk 6y ago
How does the outfit intend to fund the project? OpenAI spends millions on computing resources to train the models.
- tmalsburg2 6y agoI imagine recreating the model will be computationally cheaper because they will not have to sift through the same huge hyperparameter space as the initial GPT-3 team had to.
- thelastwave 6y agoWhy is that?
- jne2356 6y agoThis is not true. The OpenAI team only trained one full-sized GPT-3, and conducted their hyperparameter sweep on significantly smaller models (see: https://arxiv.org/abs/2001.08361 https://arxiv.org/abs/2001.08361). The compute savings from not having to do the hyperparameter sweep are negligible and do not significantly change the feasibility of the project.
- stellaathena 6y agoHey! One of the lead devs here. A cloud computing company called CoreWeave is giving us the compute for free in exchange for us releasing it. We're currently at the ~10B scale and are working on understanding datacenter scale parallelized training better, but we expect to train the model on 300-500 V100s for 4-6 months.
- jne2356 6y agoThe cloud company CoreWeave has agreed to provide the GPU resources necessary.