3 ms·
From model card: Falcon2-11B was trained on 1024 A100 40GB GPUs for the majority of the training, using a 3D parallelism strategy (TP=8, PP=1, DP=128) combined
by mmoskal 2y ago
From model card:
Falcon2-11B was trained on 1024 A100 40GB GPUs for the majority of the training, using a 3D parallelism strategy (TP=8, PP=1, DP=128) combined with ZeRO and Flash-Attention 2.
Doesn't say how long though.
- kouteiheika 2y agoIt does say how long on Huggingface: > The model training took roughly two months.