5 ms·

Please take my bug report: Dropping denormals is a huge mistake. This is easy to see if you draw out a number line for a very tiny floating-point format, for e

by burfog 7y ago

Please take my bug report: Dropping denormals is a huge mistake. This is easy to see if you draw out a number line for a very tiny floating-point format, for example with a 2-bit exponent and a 2-bit fraction. (do this on a sheet of graph paper) Without denormals, there is a huge gap surrounding zero. Strangely, the infinities were kept. Treating these as NaN is far less harmful than dropping denormals. Treating -0.0 as 0.0 and never producing -0.0 would be less harmful. (the PDF didn't say what happens) Even treating NaN values as normal numbers is probably less harmful than screwing up the denormals. IEEE floating point has lots of crazy stuff to annoy hardware vendors. Most of it isn't all that important, but denormals matter.

Veedrac 7y ago
bfloat16 is only really intended for ML purposes, and denormals don't really matter there, especially given they accurately multiply-accumulate into 32 bit floats.
- NotSammyHagar 7y ago
  Would it hurt ML usage if they added them, would it make the implementation much harder? If not, could really increase use.
  - Veedrac 7y ago
    I'm not an expert in the nitty gritty, and I've heard conflicting information, but my rough understanding is that it would be affordable and not too difficult for Xeon processors, but relatively more expensive for the dedicated Nervana neural network processors.
    - burfog 7y ago
      Sure, I can see that being the case. My point is that the Xeon should support denormals. The Bfloat16 format does allow for denormals. Intel's implementation mangles them, changing them to 0.0 on both input and output.
      - Veedrac 7y ago
        But then you need either new denormal-accepting instructions or, worse, a new global state bit enabling bfloat16 denormals, all to support use cases probably over two orders of magnitude less common than ML. What's the compelling reason to bother? Note that you need to support the denormal-disabled case because you'll want compatibility with Nervana.
        burfog 7y ago
        Intel did add a global state bit. It just isn't useful because you can't modify it. Nervana is discontinued, isn't it? Compatibility doesn't matter. It's pretty compatible anyway, as long as you aren't demanding bit-identical output.
        Veedrac 7y ago
        > Intel did add a global state bit. It just isn't useful because you can't modify it. You mean the CPUID bit? That's free. Toggling denormals isn't. > Nervana is discontinued, isn't it? Compatibility doesn't matter. It's pretty compatible anyway, as long as you aren't demanding bit-identical output. Nervana isn't discontinued according to their website[1], and bitwise compatibility does matter, certainly more than denormals do. [1] https://www.intel.ai/ai-at-ces/ https://www.intel.ai/ai-at-ces/
        burfog 7y ago
        I mean the bit to toggle denormals, not the one to identify support for the opcodes. Denormals are far more important than bitwise compatibility. To be clear, you would still be able to load a Nervana-produced number into a processor that supports denormals, and the other way would work too. You'd just avoid mangling numbers that are near zero. If you still think denormals don't matter, seriously do what I suggested: draw it out on graph paper. They matter.
        Veedrac 7y ago
        > I mean the bit to toggle denormals, not the one to identify support for the opcodes. bfloat16 doesn't handle denormals, why is there a bit to toggle it? What's it called (so I can CTRL-f for it)? > If you still think denormals don't matter, seriously do what I suggested: draw it out on graph paper. They matter. No, I get how denormals work, I know what you're pointing at. But ML genuinely doesn't care, neural nets don't give a damn about mathematical purity[1]. In contrast compatibility matters because ML doesn't give you any guarantee that it's not depending on the behaviour at small values, and minor differences in rounding does cause issues. For example, Leela Chess Zero had difficulties with reproducibility because different GPUs round floats differently. [1] Fun but relevant aside: https://openai.com/blog/nonlinear-computation-in-linear-networks/ https://openai.com/blog/nonlinear-computation-in-linear-netw...

by burfog 7y ago

Veedrac 7y ago

bfloat16 is only really intended for ML purposes, and denormals don't really matter there, especially given they accurately multiply-accumulate into 32 bit floats.

NotSammyHagar 7y ago
Would it hurt ML usage if they added them, would it make the implementation much harder? If not, could really increase use.
- Veedrac 7y ago
  I'm not an expert in the nitty gritty, and I've heard conflicting information, but my rough understanding is that it would be affordable and not too difficult for Xeon processors, but relatively more expensive for the dedicated Nervana neural network processors.
  - burfog 7y ago
    Sure, I can see that being the case. My point is that the Xeon should support denormals. The Bfloat16 format does allow for denormals. Intel's implementation mangles them, changing them to 0.0 on both input and output.
    - Veedrac 7y ago
      But then you need either new denormal-accepting instructions or, worse, a new global state bit enabling bfloat16 denormals, all to support use cases probably over two orders of magnitude less common than ML. What's the compelling reason to bother? Note that you need to support the denormal-disabled case because you'll want compatibility with Nervana.
      - burfog 7y ago
        Intel did add a global state bit. It just isn't useful because you can't modify it. Nervana is discontinued, isn't it? Compatibility doesn't matter. It's pretty compatible anyway, as long as you aren't demanding bit-identical output.
        Veedrac 7y ago
        > Intel did add a global state bit. It just isn't useful because you can't modify it. You mean the CPUID bit? That's free. Toggling denormals isn't. > Nervana is discontinued, isn't it? Compatibility doesn't matter. It's pretty compatible anyway, as long as you aren't demanding bit-identical output. Nervana isn't discontinued according to their website[1], and bitwise compatibility does matter, certainly more than denormals do. [1] https://www.intel.ai/ai-at-ces/ https://www.intel.ai/ai-at-ces/
        burfog 7y ago
        I mean the bit to toggle denormals, not the one to identify support for the opcodes. Denormals are far more important than bitwise compatibility. To be clear, you would still be able to load a Nervana-produced number into a processor that supports denormals, and the other way would work too. You'd just avoid mangling numbers that are near zero. If you still think denormals don't matter, seriously do what I suggested: draw it out on graph paper. They matter.
        Veedrac 7y ago
        > I mean the bit to toggle denormals, not the one to identify support for the opcodes. bfloat16 doesn't handle denormals, why is there a bit to toggle it? What's it called (so I can CTRL-f for it)? > If you still think denormals don't matter, seriously do what I suggested: draw it out on graph paper. They matter. No, I get how denormals work, I know what you're pointing at. But ML genuinely doesn't care, neural nets don't give a damn about mathematical purity[1]. In contrast compatibility matters because ML doesn't give you any guarantee that it's not depending on the behaviour at small values, and minor differences in rounding does cause issues. For example, Leela Chess Zero had difficulties with reproducibility because different GPUs round floats differently. [1] Fun but relevant aside: https://openai.com/blog/nonlinear-computation-in-linear-networks/ https://openai.com/blog/nonlinear-computation-in-linear-netw...