Traditionally Deep Learning frameworks were all using FP32. With FP16 one can theoretically get 2x speed and 2x larger models with the same VRAM capacity. For

Traditionally Deep Learning frameworks were all using FP32. With FP16 one can theoretically get 2x speed and 2x larger models with the same VRAM capacity. For inferencing with INT8/INT4 it can be even way better (good for embedded stuff). The downside is that sometimes more complex/deep models don't converge (or converge less often than FP32). Sometimes there are framework issues with some advanced FP16 stuff.