jne2356

3 ms·

1. 1.▲
   by jne2356 6y ago

   The resources for the GPT-3 replication will be provided by the cloud company CoreWeave.
2. 2.▲
   by jne2356 6y ago

   They get this suggestion a lot. There's a section in their FAQ that explains why it's infeasible. https://github.com/EleutherAI/info
3. 3.▲
   by jne2356 6y ago

   OpenAI only trained the full sized GPT-3 once. Hyperparameter sweep was conducted on significantly smaller models (see: https://arxiv.org/abs/2001.08361 )
4. 4.▲
   by jne2356 6y ago

   This is not true. The OpenAI team only trained one full-sized GPT-3, and conducted their hyperparameter sweep on significantly smaller models (see: https://arxiv.org/abs/2001.08361 ). The compute savings from not having
5. 5.▲
   by jne2356 6y ago

   EleutherAI has already secured the resources necessary. They get the seti@home suggestion a lot. There's a section in their FAQ that explains why it's infeasible. https://github.com/EleutherAI/info
6. 6.▲
   by jne2356 6y ago

   The cloud company CoreWeave has agreed to provide the GPU resources necessary.
7. 7.▲
   by jne2356 6y ago

   GPT-3 does not fit in any one GPU that exists at present. It's already spread out across multiple GPUs.
8. 8.▲
   by jne2356 6y ago

   They get this suggestion a lot. There's a section in their FAQ that explains why it's infeasible. https://github.com/EleutherAI/info