4 ms·

Not to defend Intel because I think Intel has become a Ship of Fools With a lot of money, but AMD has FP24, Nvidia has FP16 and Google has TPU 2 with an FP16 im

Not to defend Intel because I think Intel has become a Ship of Fools With a lot of money, but AMD has FP24, Nvidia has FP16 and Google has TPU 2 with an FP16 implementation so proprietary they can't talk about it. I think the valid concern here is that if this doesn't take off immediately they will end-of-life it and not so much the floating point format.

borramakot 9y ago
I won't defend FP24, but FP16 is a widely supported IEEE standard. https://en.wikipedia.org/wiki/IEEE_754 https://en.wikipedia.org/wiki/IEEE_754
- dogma1138 9y ago
  Not exactly, IEEE 754 defines very specific rules (e.g. what operations can be done, order of operations and how the results are handled), it's basically exists so that if the same calculation is done on different hardware the result will be the same (or within a given margin of error since even IEEE 754 isn't 100% strict). For the most part when you have hardware that supports one of the binary standards say FP32 you'll have a compliance sheet, more often than not it will not be 100% compliant for example NVIDIA GPUs were not IEEE 754 compliant when Tesla came out, 2nd Generation Tesla cards were FMA compliant, div and sqrt operations were not IEEE 754 compliant until Fermi. So the GP was correct, back in the old DX9 days when AMD went with FP24 and NVIDIA with FP16/32 both implementations were proprietary, and this is still the case, they just often offer a 754 compliant mode. You can disable 754 compliance to run faster (sometimes considerably so) e.g. in CUDA you can use the NVCC flag --use_fast_math to do so.
- justincormack 9y ago
  FP16 is an interchange format in IEEE not a computer format like it is being used here. The idea is you can compress for storage into 16 bits
deepnotderp 9y ago
Ah, someone who agrees with me that Intel has more money than sense :) But yes, I agree, that criticism is stupid. In fact, I would argue that if you use the IEEE standard for a deep learning processor, you're the one that's stupid.

borramakot 9y ago

I won't defend FP24, but FP16 is a widely supported IEEE standard. https://en.wikipedia.org/wiki/IEEE_754 https://en.wikipedia.org/wiki/IEEE_754

dogma1138 9y ago
Not exactly, IEEE 754 defines very specific rules (e.g. what operations can be done, order of operations and how the results are handled), it's basically exists so that if the same calculation is done on different hardware the result will be the same (or within a given margin of error since even IEEE 754 isn't 100% strict). For the most part when you have hardware that supports one of the binary standards say FP32 you'll have a compliance sheet, more often than not it will not be 100% compliant for example NVIDIA GPUs were not IEEE 754 compliant when Tesla came out, 2nd Generation Tesla cards were FMA compliant, div and sqrt operations were not IEEE 754 compliant until Fermi. So the GP was correct, back in the old DX9 days when AMD went with FP24 and NVIDIA with FP16/32 both implementations were proprietary, and this is still the case, they just often offer a 754 compliant mode. You can disable 754 compliance to run faster (sometimes considerably so) e.g. in CUDA you can use the NVCC flag --use_fast_math to do so.
justincormack 9y ago
FP16 is an interchange format in IEEE not a computer format like it is being used here. The idea is you can compress for storage into 16 bits

deepnotderp 9y ago

Ah, someone who agrees with me that Intel has more money than sense :) But yes, I agree, that criticism is stupid. In fact, I would argue that if you use the IEEE standard for a deep learning processor, you're the one that's stupid.