5 ms·
Using mixed precision training you can do most operations in fp16 and just a few in fp32 where it's needed. This is the norm for NVIDIA GPU training nowadays. F
by jph00 4y ago
Using mixed precision training you can do most operations in fp16 and just a few in fp32 where it's needed. This is the norm for NVIDIA GPU training nowadays. For instance using fastai add `.to_fp16()` after your learner call, and that happens automatically.
- omegalulw 4y agoHow is the choice between fp16 and fp32 made? Is it like if any gradients in the tensor need the extra range you use fp32?
- h-jones 4y agoThe PyTorch docs give a pretty good overview of AMP here https://pytorch.org/tutorials/recipes/recipes/amp_recipe.html https://pytorch.org/tutorials/recipes/recipes/amp_recipe.htm... and an overview of which operations cast to which dtype can be found here https://pytorch.org/docs/stable/amp.html#autocast-op-reference https://pytorch.org/docs/stable/amp.html#autocast-op-referen.... Edit: Fixed second link.
- andoma 4y agoThis article [0] from Nvidia gives a good overview of how mixed precision training works. Super high level (from section 3): 1. Converting the model to use the float16 data type where possible. 2. Keeping float32 master weights to accumulate per-iteration weight updates. 3. Using loss scaling to preserve small gradient values. [0] https://docs.nvidia.com/deeplearning/performance/mixed-precision-training/index.html https://docs.nvidia.com/deeplearning/performance/mixed-preci...