3 ms·
They offload parameters, gradients and optimizer states (such as moment, velocity and exponential avg of these in Adam) into CPU memory.
by liuliu 6y ago
They offload parameters, gradients and optimizer states (such as moment, velocity and exponential avg of these in Adam) into CPU memory.
- p1esk 6y agoThey did all that before: https://arxiv.org/abs/2101.06840 https://arxiv.org/abs/2101.06840, but they could only fit a model with 13B weights on a single V100.