3 ms·

Subnormal numbers have a different, basically fixed-point, representation. They exist in order to bridge the large (relatively speaking; indeed "infinite" in a

Subnormal numbers have a different, basically fixed-point, representation. They exist in order to bridge the large (relatively speaking; indeed "infinite" in a sense) gap between the least positive normal number, zero, and the greatest negative normal number, caused by the usual significand-exponent representation. Most "mundane" uses of floating point have no need for subnormal numbers, and numbers that underflow could just be flushed to zero. But they’re sometimes important in scientific computing to ensure sufficient smoothness around zero, avoiding precision issues.

bee_rider 8d ago
I don’t know how useful they are in scientific computing either, really. They are less precise than normalized numbers… if flushing them makes a difference I think it is a bad algorithm smell.
- adrian_b 7d ago
  Scientific computing can be done only in 2 ways, either with subnormals or by enabling the underflow exception and writing a suitable exception handler for it. If the use of subnormals is disabled with FTZ/DAZ that is guaranteed to generate big errors and it is completely unpredictable how big the errors will be. If a computational algorithm generates underflows at some place, there is no way to modify the algorithm so that flushing-to-zero will not make any difference (i.e. no errors). What is possible, is to modify the algorithm so that underflows will never happen. This was the traditional way of writing numeric algorithms. Because on early computers underflows would crash the program, the same as overflows, one had to improve the algorithm in order to avoid both underflows and overflows. Subnormals and infinities have been introduced in the standard precisely for lazier programmers, so that they would be able to avoid the rewriting of algorithms without the risks that underflows and overflows would generate major errors. Unfortunately, it seems that for some programmers this is still not enough, because they want simultaneously to not be bothered with rewriting the algorithms and to have the program run as fast as with an optimized algorithm. For this, the solution is very simple and it is not enabling FTZ/DAZ, which unless is done for a game might cause unpredictable financial losses for an unsuspecting customer, who expects that a computer must provide correct results. The right solution is to not buy Intel CPUs or any other kind of processors whose vendor believes that the correctness of computations does not matter. It should be noted however, that the Intel server CPUs use CPU cores that are obsolete in desktop and laptop CPUs, i.e. the tested Intel CPUs use cores like those in Meteor Lake and Raptor Lake CPUs. I do not know if the more recent Intel CPU cores, from Panther Lake/Arrow Lake S/Arrow Lake H/Lunar Lake, have retained this Intel misfeature, which has characterized the Intel CPUs for much more than a decade. If someone says that they have enabled FTZ/DAZ and they did not see any significant difference in the results of a program, that is complete B*S*T, because it is impossible to test exhaustively any program that does floating-point computations and the errors are expected to happen only for certain values, which are unlikely to be encountered during testing, but you cannot predict that those values will not be encountered in production.
nayuki 8d ago
I can think of one useful property of subnormal numbers off the top of my head. If subnormal processing is enabled, then for all finite values of `a` and `b`, `a != b` if and only if `a - b != 0`. But if subnormals are flushed to zero, then two tiny normal distinct values `a` and `b` would have a subnormal difference that is flushed to zero.
- bryanlarsen 8d ago
  Isn't that just a scale issue that exists with or without subnormals? If a and b are closer to zero than the smallest representable number, a and b compare as the same. With subnormals your smallest possible number is smaller than without, but it's still the same issue.
  - Sharlin 8d ago
    Because subnormals are fixed point, ie. have a fixed exponent, the difference of any two distinct subnormal values is nonzero like with integers.
    - bryanlarsen 8d ago
      But that same statement applies to normal values too, right? With normal numbers you might get the oddity of a-b -> a even if b is nonzero, but you don't get the oddity of a-b -> 0 unless the same number is represented, IIUC. A and B might not be bit identical, but they represent the same number if the difference is 0.
jwmerrill 8d ago
One nice thing that subnormals get you is the property that if x-y == 0 then x == y. If you want to guard against division by 0, and your denominator is a difference of two terms, it’s nice to be able to check equality of those terms and know that if they are not equal, then their difference will not be 0. More generally, subnormals are needed for Sterbenz Lemma to hold everywhere: https://en.wikipedia.org/wiki/Sterbenz_lemma https://en.wikipedia.org/wiki/Sterbenz_lemma
adrian_b 7d ago
On early computers, any underflow generated an exception that would crash the program if not handled. This was very good, because underflows completely break the assumptions about floating-point arithmetic on which numeric algorithms are based, so the errors in the final results become unpredictable. Subnormal numbers have been introduced as a means to avoid handling every underflow exception, because typically the use of subnormals eliminates the errors that would otherwise be caused by underflows. The flush-to-zero and denormals-of-zero options must be strictly forbidden for any general-purpose applications. They should be permitted only in applications where there is no doubt that regardless how big the errors will be they will not have any really harmful effect, which is true for games and perhaps for AI, but for little else. This is another great misfeature promoted by Intel, in order to win meaningless benchmarks. It would have been much better if these standard-breaking features would not have existed, because they are much more often used when they should not be used, than when they are harmless.

bee_rider 8d ago

I don’t know how useful they are in scientific computing either, really. They are less precise than normalized numbers… if flushing them makes a difference I think it is a bad algorithm smell.

adrian_b 7d ago
Scientific computing can be done only in 2 ways, either with subnormals or by enabling the underflow exception and writing a suitable exception handler for it. If the use of subnormals is disabled with FTZ/DAZ that is guaranteed to generate big errors and it is completely unpredictable how big the errors will be. If a computational algorithm generates underflows at some place, there is no way to modify the algorithm so that flushing-to-zero will not make any difference (i.e. no errors). What is possible, is to modify the algorithm so that underflows will never happen. This was the traditional way of writing numeric algorithms. Because on early computers underflows would crash the program, the same as overflows, one had to improve the algorithm in order to avoid both underflows and overflows. Subnormals and infinities have been introduced in the standard precisely for lazier programmers, so that they would be able to avoid the rewriting of algorithms without the risks that underflows and overflows would generate major errors. Unfortunately, it seems that for some programmers this is still not enough, because they want simultaneously to not be bothered with rewriting the algorithms and to have the program run as fast as with an optimized algorithm. For this, the solution is very simple and it is not enabling FTZ/DAZ, which unless is done for a game might cause unpredictable financial losses for an unsuspecting customer, who expects that a computer must provide correct results. The right solution is to not buy Intel CPUs or any other kind of processors whose vendor believes that the correctness of computations does not matter. It should be noted however, that the Intel server CPUs use CPU cores that are obsolete in desktop and laptop CPUs, i.e. the tested Intel CPUs use cores like those in Meteor Lake and Raptor Lake CPUs. I do not know if the more recent Intel CPU cores, from Panther Lake/Arrow Lake S/Arrow Lake H/Lunar Lake, have retained this Intel misfeature, which has characterized the Intel CPUs for much more than a decade. If someone says that they have enabled FTZ/DAZ and they did not see any significant difference in the results of a program, that is complete B*S*T, because it is impossible to test exhaustively any program that does floating-point computations and the errors are expected to happen only for certain values, which are unlikely to be encountered during testing, but you cannot predict that those values will not be encountered in production.

nayuki 8d ago

I can think of one useful property of subnormal numbers off the top of my head. If subnormal processing is enabled, then for all finite values of `a` and `b`, `a != b` if and only if `a - b != 0`. But if subnormals are flushed to zero, then two tiny normal distinct values `a` and `b` would have a subnormal difference that is flushed to zero.

bryanlarsen 8d ago
Isn't that just a scale issue that exists with or without subnormals? If a and b are closer to zero than the smallest representable number, a and b compare as the same. With subnormals your smallest possible number is smaller than without, but it's still the same issue.
- Sharlin 8d ago
  Because subnormals are fixed point, ie. have a fixed exponent, the difference of any two distinct subnormal values is nonzero like with integers.
  - bryanlarsen 8d ago
    But that same statement applies to normal values too, right? With normal numbers you might get the oddity of a-b -> a even if b is nonzero, but you don't get the oddity of a-b -> 0 unless the same number is represented, IIUC. A and B might not be bit identical, but they represent the same number if the difference is 0.

jwmerrill 8d ago

One nice thing that subnormals get you is the property that if x-y == 0 then x == y. If you want to guard against division by 0, and your denominator is a difference of two terms, it’s nice to be able to check equality of those terms and know that if they are not equal, then their difference will not be 0. More generally, subnormals are needed for Sterbenz Lemma to hold everywhere: https://en.wikipedia.org/wiki/Sterbenz_lemma https://en.wikipedia.org/wiki/Sterbenz_lemma

adrian_b 7d ago

On early computers, any underflow generated an exception that would crash the program if not handled. This was very good, because underflows completely break the assumptions about floating-point arithmetic on which numeric algorithms are based, so the errors in the final results become unpredictable. Subnormal numbers have been introduced as a means to avoid handling every underflow exception, because typically the use of subnormals eliminates the errors that would otherwise be caused by underflows. The flush-to-zero and denormals-of-zero options must be strictly forbidden for any general-purpose applications. They should be permitted only in applications where there is no doubt that regardless how big the errors will be they will not have any really harmful effect, which is true for games and perhaps for AI, but for little else. This is another great misfeature promoted by Intel, in order to win meaningless benchmarks. It would have been much better if these standard-breaking features would not have existed, because they are much more often used when they should not be used, than when they are harmless.