7 ms·

> To maintain integrity and prevent misuse, we are releasing our model under a noncommercial license focused on research use cases. if say i wanted to replicat

by swyx 4y ago

> To maintain integrity and prevent misuse, we are releasing our model under a noncommercial license focused on research use cases. if say i wanted to replicate this paper for commercial use, what would it take and how do i get started? would FB have a basis for objection?

minimaxir 4y ago
A metric-ton of GPU compute, for starters.
chandureddyvari 4y ago
82342 GPU(A-100 80GB) hours for training their 7B(smallest) model according to their paper.
- riku_iki 4y ago
  115 GPUs can be used to train it within month.
causalmodels 4y ago
I'm going to assume you know how to stand up and manage a distributed training cluster as a simplifying assumption. Note this is an aggressive assumption. You would need to replicate the preprocessing steps. Replicating these steps is going to be tricky as they are not described in detail.Then you would need to implement the model using xformers [1]. Using xformers is going to save you a lot of compute spend. You will need to manually implement the backwards pass to reduce recomputation of expensive activations. The model was trained using 2048 A100 GPUs with 80GBs of VRAM. A single 8 A100 GPU machine from Lambda Cloud costs $12.00/hr [2]. The team from meta used 256 such machines giving you a per day cost of $73,728. It takes 21 days to train this model. The upfront lower bound cost estimate of doing this is [(12.00 * 24) * 21 * 256) = ] $1,548,288 dollars assuming everything goes smoothly and your model doesn't bite it during training. You may be able to negotiate bulk pricing for these types of workloads. That dollar value is just for the compute resources alone. Given the compute costs required you will probably also want a team composed of ML Ops engineers to monitor the training cluster and research scientists to help you with the preprocessing and model pipelines. [1] https://github.com/facebookresearch/xformers https://github.com/facebookresearch/xformers [2] https://lambdalabs.com/service/gpu-cloud https://lambdalabs.com/service/gpu-cloud edit: these costs are for the 65B parameter model.
- swyx 4y ago
  thank you! love getting a peek into infra math. hopefully these will come down someday but also my resources rise up to be able to train these inhouse. can foresee foundation model training being a core competency of many tech platform companies in future.

by swyx 4y ago

minimaxir 4y ago

A metric-ton of GPU compute, for starters.

chandureddyvari 4y ago

82342 GPU(A-100 80GB) hours for training their 7B(smallest) model according to their paper.

riku_iki 4y ago
115 GPUs can be used to train it within month.

causalmodels 4y ago

I'm going to assume you know how to stand up and manage a distributed training cluster as a simplifying assumption. Note this is an aggressive assumption. You would need to replicate the preprocessing steps. Replicating these steps is going to be tricky as they are not described in detail.Then you would need to implement the model using xformers [1]. Using xformers is going to save you a lot of compute spend. You will need to manually implement the backwards pass to reduce recomputation of expensive activations. The model was trained using 2048 A100 GPUs with 80GBs of VRAM. A single 8 A100 GPU machine from Lambda Cloud costs $12.00/hr [2]. The team from meta used 256 such machines giving you a per day cost of $73,728. It takes 21 days to train this model. The upfront lower bound cost estimate of doing this is [(12.00 * 24) * 21 * 256) = ] $1,548,288 dollars assuming everything goes smoothly and your model doesn't bite it during training. You may be able to negotiate bulk pricing for these types of workloads. That dollar value is just for the compute resources alone. Given the compute costs required you will probably also want a team composed of ML Ops engineers to monitor the training cluster and research scientists to help you with the preprocessing and model pipelines. [1] https://github.com/facebookresearch/xformers https://github.com/facebookresearch/xformers [2] https://lambdalabs.com/service/gpu-cloud https://lambdalabs.com/service/gpu-cloud edit: these costs are for the 65B parameter model.

swyx 4y ago
thank you! love getting a peek into infra math. hopefully these will come down someday but also my resources rise up to be able to train these inhouse. can foresee foundation model training being a core competency of many tech platform companies in future.