On AMD, if I'm not not mistaken, FP16 is twice of FP32, as it should be "For their consumer cards, NVIDIA has severely limited FP16 CUDA performance. GTX 1080’

On AMD, if I'm not not mistaken, FP16 is twice of FP32, as it should be "For their consumer cards, NVIDIA has severely limited FP16 CUDA performance. GTX 1080’s FP16 instruction rate is 1/128th its FP32 instruction rate, or after you factor in vec2 packing, the resulting theoretical performance (in FLOPs) is 1/64th the FP32 rate, or about 138 GFLOPs." https://www.anandtech.com/show/10325/the-nvidia-geforce-gtx-1080-and-1070-founders-edition-review/5 https://www.anandtech.com/show/10325/the-nvidia-geforce-gtx-... "FP16 performance is 1/64th and FP64 is 1/32th of FP32 performance." https://blog.inten.to/hardware-for-deep-learning-part-3-gpu-8906c1644664 https://blog.inten.to/hardware-for-deep-learning-part-3-gpu-...