3 ms·
GP could be mentioning quantization aware training, during which the weight and gradient are still computed in fp16/fp32.
by rfoo 4y ago
GP could be mentioning quantization aware training, during which the weight and gradient are still computed in fp16/fp32.
- CuriouslyC 4y agoIt can go farther than that, it seems like the weight gradients are the main thing where the precision is a bottleneck (see https://arxiv.org/abs/1805.11046 https://arxiv.org/abs/1805.11046).