4 ms·
But it's not clear how they managed to improve training on a single GPU: they say they can fit 40B model on a single V100.
by p1esk 6y ago
But it's not clear how they managed to improve training on a single GPU: they say they can fit 40B model on a single V100.
- liuliu 6y agoThey offload parameters, gradients and optimizer states (such as moment, velocity and exponential avg of these in Adam) into CPU memory.
- p1esk 6y agoThey did all that before: https://arxiv.org/abs/2101.06840 https://arxiv.org/abs/2101.06840, but they could only fit a model with 13B weights on a single V100.