4 ms·

Flushing denormals to zero only matters if your calculations are already running into the lower end of floating-point exponents (and even with denormals, if the

Flushing denormals to zero only matters if your calculations are already running into the lower end of floating-point exponents (and even with denormals, if they're doing that, they're going to run into lost precision anyway sooner or later). The useful thing denormals do is make the loss of precision at that point gradual, instead of sudden. But you're still losing precision, and a few orders of magnitude later you're going to wind up at 0 either way. If your formulas are producing intermediate results with values that run into the lower end of FP range, and it matters that those values retain precision there, then you're either using the wrong FP type or you're using the wrong formulas. Your code is likely already broken, the breakage is just rarer than in flush-to-zero mode. So just enable FTZ mode, and if you run into issues, you need to fix that code (e.g. don't divide by values that can be too close to 0), not switch denormals on.

adrian_b 5y ago
Your arguments are correct, but the conclusion does not result from them. If we assume that underflows happen in your program and this, as you say, is a sign that greater problems will be caused by that, then you must not enable flush-to-zero, but you must enable trap-on-underflow, to see where underflows happen and to investigate the reason and maybe rearrange your formulas to avoid the too small results. Flush-to-zero may sometimes lead to crashes, when it becomes obvious that something is wrong, but more often you just get some results with large errors that are difficult to distinguish from good results. Opinions obviously vary, but I have never seen any good use case for flush-to-zero. Underflows are either so rare that their performance does not matter, or if they are so frequent that flush-to-zero will increase the speed, then your code has a problem and the formulas should be restructured, which will both increase the speed and eliminate the loss of precision.
- marcan_42 5y ago
  It is normal to get values close to zero in a correctly implemented floating point algorithm - what matters is that, at that point, you don't rely on their dynamic range or precision. Trap-on-underflow would produce many false positives for algorithms which would be perfectly happy having those numbers be treated as zero. This is not uncommon - it is completely expected with, say, anything implementing an exponential decay! A slow exponential decay could run into subnormals for large spans of time, yet be fine having those treated as zero.
- stephencanon 5y ago
  > Opinions obviously vary, but I have never seen any good use case for flush-to-zero. The classical use case is real-time audio, where an IIR filter may have quite slow decay, such that the signal stays in the subnormal regime for many samples. If this happens and your hardware has significant stalls for subnormal data, you may miss your real-time deadline resulting in clicking or other audio corruption.
  - adrian_b 5y ago
    I agree that this is a good example and there are probably similar cases in video processing and in graphics rendering. However not all CPUs stall for subnormal data and if the CPU designers would have ensured that none of them stall, no such discussions would have been ever needed. Outside such special cases like real-time DSP and graphics, using flush-to-zero or options like -Ofast or -ffast-math are seldom justified and they can be dangerous, especially when they are used without a prior analysis and tests to determine whether exception conditions really appear when the program is run, and why.
    - stephencanon 5y ago
      > However not all CPUs stall for subnormal data and if the CPU designers would have ensured that none of them stall, no such discussions would have been ever needed. I don't really disagree (I've pushed hard for no subnormal stalls for years), but there are some timing and area tradeoffs that come with this that make it a somewhat hard sell for FPUs that aren't built entirely around FMA (there are tradeoffs even for designs that are purely FMA based, but they're a lot smaller). Mainstream x86 designs _still_ have some subnormal stalls!
    - nyanpasu64 5y ago
      > Outside such special cases like real-time DSP and graphics Personally I see audio and graphics as the common case, and scientific computing as the special case ;) Do game physics engines expect denormals to be preserved?

adrian_b 5y ago

Your arguments are correct, but the conclusion does not result from them. If we assume that underflows happen in your program and this, as you say, is a sign that greater problems will be caused by that, then you must not enable flush-to-zero, but you must enable trap-on-underflow, to see where underflows happen and to investigate the reason and maybe rearrange your formulas to avoid the too small results. Flush-to-zero may sometimes lead to crashes, when it becomes obvious that something is wrong, but more often you just get some results with large errors that are difficult to distinguish from good results. Opinions obviously vary, but I have never seen any good use case for flush-to-zero. Underflows are either so rare that their performance does not matter, or if they are so frequent that flush-to-zero will increase the speed, then your code has a problem and the formulas should be restructured, which will both increase the speed and eliminate the loss of precision.

marcan_42 5y ago
It is normal to get values close to zero in a correctly implemented floating point algorithm - what matters is that, at that point, you don't rely on their dynamic range or precision. Trap-on-underflow would produce many false positives for algorithms which would be perfectly happy having those numbers be treated as zero. This is not uncommon - it is completely expected with, say, anything implementing an exponential decay! A slow exponential decay could run into subnormals for large spans of time, yet be fine having those treated as zero.
stephencanon 5y ago
> Opinions obviously vary, but I have never seen any good use case for flush-to-zero. The classical use case is real-time audio, where an IIR filter may have quite slow decay, such that the signal stays in the subnormal regime for many samples. If this happens and your hardware has significant stalls for subnormal data, you may miss your real-time deadline resulting in clicking or other audio corruption.
- adrian_b 5y ago
  I agree that this is a good example and there are probably similar cases in video processing and in graphics rendering. However not all CPUs stall for subnormal data and if the CPU designers would have ensured that none of them stall, no such discussions would have been ever needed. Outside such special cases like real-time DSP and graphics, using flush-to-zero or options like -Ofast or -ffast-math are seldom justified and they can be dangerous, especially when they are used without a prior analysis and tests to determine whether exception conditions really appear when the program is run, and why.
  - stephencanon 5y ago
    > However not all CPUs stall for subnormal data and if the CPU designers would have ensured that none of them stall, no such discussions would have been ever needed. I don't really disagree (I've pushed hard for no subnormal stalls for years), but there are some timing and area tradeoffs that come with this that make it a somewhat hard sell for FPUs that aren't built entirely around FMA (there are tradeoffs even for designs that are purely FMA based, but they're a lot smaller). Mainstream x86 designs _still_ have some subnormal stalls!
  - nyanpasu64 5y ago
    > Outside such special cases like real-time DSP and graphics Personally I see audio and graphics as the common case, and scientific computing as the special case ;) Do game physics engines expect denormals to be preserved?