5 ms·

ISA: https://software.intel.com/sites/default/files/managed/c5/15/architecture-instruction-set-extensions-programming-reference.pdf https://software.intel.com/s

ISA: https://software.intel.com/sites/default/files/managed/c5/15/architecture-instruction-set-extensions-programming-reference.pdf https://software.intel.com/sites/default/files/managed/c5/15... Look for anything marked with AVX512_BF16 CPUID feature flag. Numerical details: https://software.intel.com/sites/default/files/managed/40/8b/bf16-hardware-numerics-definition-white-paper.pdf https://software.intel.com/sites/default/files/managed/40/8b... Support for bfloat16 is already present in MKL-DNN (https://github.com/intel/mkl-dnn https://github.com/intel/mkl-dnn) Disclaimer: I work for Intel

burfog 7y ago
Please take my bug report: Dropping denormals is a huge mistake. This is easy to see if you draw out a number line for a very tiny floating-point format, for example with a 2-bit exponent and a 2-bit fraction. (do this on a sheet of graph paper) Without denormals, there is a huge gap surrounding zero. Strangely, the infinities were kept. Treating these as NaN is far less harmful than dropping denormals. Treating -0.0 as 0.0 and never producing -0.0 would be less harmful. (the PDF didn't say what happens) Even treating NaN values as normal numbers is probably less harmful than screwing up the denormals. IEEE floating point has lots of crazy stuff to annoy hardware vendors. Most of it isn't all that important, but denormals matter.
- Veedrac 7y ago
  bfloat16 is only really intended for ML purposes, and denormals don't really matter there, especially given they accurately multiply-accumulate into 32 bit floats.
  - NotSammyHagar 7y ago
    Would it hurt ML usage if they added them, would it make the implementation much harder? If not, could really increase use.
    - Veedrac 7y ago
      I'm not an expert in the nitty gritty, and I've heard conflicting information, but my rough understanding is that it would be affordable and not too difficult for Xeon processors, but relatively more expensive for the dedicated Nervana neural network processors.
      - burfog 7y ago
        Sure, I can see that being the case. My point is that the Xeon should support denormals. The Bfloat16 format does allow for denormals. Intel's implementation mangles them, changing them to 0.0 on both input and output.
        Veedrac 7y ago
        But then you need either new denormal-accepting instructions or, worse, a new global state bit enabling bfloat16 denormals, all to support use cases probably over two orders of magnitude less common than ML. What's the compelling reason to bother? Note that you need to support the denormal-disabled case because you'll want compatibility with Nervana.
        burfog 7y ago
        Intel did add a global state bit. It just isn't useful because you can't modify it. Nervana is discontinued, isn't it? Compatibility doesn't matter. It's pretty compatible anyway, as long as you aren't demanding bit-identical output.
        Veedrac 7y ago
        > Intel did add a global state bit. It just isn't useful because you can't modify it. You mean the CPUID bit? That's free. Toggling denormals isn't. > Nervana is discontinued, isn't it? Compatibility doesn't matter. It's pretty compatible anyway, as long as you aren't demanding bit-identical output. Nervana isn't discontinued according to their website[1], and bitwise compatibility does matter, certainly more than denormals do. [1] https://www.intel.ai/ai-at-ces/ https://www.intel.ai/ai-at-ces/

burfog 7y ago

Please take my bug report: Dropping denormals is a huge mistake. This is easy to see if you draw out a number line for a very tiny floating-point format, for example with a 2-bit exponent and a 2-bit fraction. (do this on a sheet of graph paper) Without denormals, there is a huge gap surrounding zero. Strangely, the infinities were kept. Treating these as NaN is far less harmful than dropping denormals. Treating -0.0 as 0.0 and never producing -0.0 would be less harmful. (the PDF didn't say what happens) Even treating NaN values as normal numbers is probably less harmful than screwing up the denormals. IEEE floating point has lots of crazy stuff to annoy hardware vendors. Most of it isn't all that important, but denormals matter.

Veedrac 7y ago
bfloat16 is only really intended for ML purposes, and denormals don't really matter there, especially given they accurately multiply-accumulate into 32 bit floats.
- NotSammyHagar 7y ago
  Would it hurt ML usage if they added them, would it make the implementation much harder? If not, could really increase use.
  - Veedrac 7y ago
    I'm not an expert in the nitty gritty, and I've heard conflicting information, but my rough understanding is that it would be affordable and not too difficult for Xeon processors, but relatively more expensive for the dedicated Nervana neural network processors.
    - burfog 7y ago
      Sure, I can see that being the case. My point is that the Xeon should support denormals. The Bfloat16 format does allow for denormals. Intel's implementation mangles them, changing them to 0.0 on both input and output.
      - Veedrac 7y ago
        But then you need either new denormal-accepting instructions or, worse, a new global state bit enabling bfloat16 denormals, all to support use cases probably over two orders of magnitude less common than ML. What's the compelling reason to bother? Note that you need to support the denormal-disabled case because you'll want compatibility with Nervana.
        burfog 7y ago
        Intel did add a global state bit. It just isn't useful because you can't modify it. Nervana is discontinued, isn't it? Compatibility doesn't matter. It's pretty compatible anyway, as long as you aren't demanding bit-identical output.
        Veedrac 7y ago
        > Intel did add a global state bit. It just isn't useful because you can't modify it. You mean the CPUID bit? That's free. Toggling denormals isn't. > Nervana is discontinued, isn't it? Compatibility doesn't matter. It's pretty compatible anyway, as long as you aren't demanding bit-identical output. Nervana isn't discontinued according to their website[1], and bitwise compatibility does matter, certainly more than denormals do. [1] https://www.intel.ai/ai-at-ces/ https://www.intel.ai/ai-at-ces/