3 ms·
You can't even train BERT_large on a 12/16GB GPU, and on a single 15TFlops GPU it might take a year to train. GPUs are too slow :-(
by bitL 8y ago
You can't even train BERT_large on a 12/16GB GPU, and on a single 15TFlops GPU it might take a year to train. GPUs are too slow :-(
- riku_iki 8y agoTPU is also slow, they used pod with 64 TPUs for training BERT. You probably can achieve similar result using distributed training on multiple GPU machines.
- solomatov 8y agoYou can but it will be really slow. You can load just parts of the model, and store them on the disk/in memory :-)