5 ms·

> I will note that the author says there are 1,056,964,610 "normal floating-point numbers" in [0,1) and I am not entirely sure why this disagrees with the numbe

by mkl 2y ago

> I will note that the author says there are 1,056,964,610 "normal floating-point numbers" in [0,1) and I am not entirely sure why this disagrees with the number I produced. The reason is there in the quote. You counted both normal and subnormal [1] floating point numbers, and they just counted normal floating point numbers. On mobile so can't check precise details, but that would explain it. [1] https://en.wikipedia.org/wiki/Subnormal_number https://en.wikipedia.org/wiki/Subnormal_number

tialaramex 2y ago
Huh, that Wiki link taught me something I didn't know - I hadn't realised a different FP representation could have Denormals (a word I knew) which weren't Subnormal (a new word to me, more precisely defining what's different about these values). The Subnormals (presumably any Denormals regardless?) tend to be annoyingly slow, because the FP hardware usually special cases them to some slow but correct silicon not a fast path for the usual case. So, in some code it can make sense to "zap" subnormals, converting them to zero since they're almost zero anyway. Obviously in other code this is a terrible idea, so your domain knowledge comes into play when authoring such code. For example if we're using -1 .. +1 as the 0dB for our Karaoke software, a subnormal is clearly inaudible noise, but it might make our sophisticated reverb simulator (so you can pretend you're singing in a big auditorium) a thousand times slower than normal, so, zap those subnormals before the simulator and we hear the same results but don't get punished with miserable performance.
- mkl 2y ago
  Denormal and subnormal are synonyms as far as I know and can find. That first sentence is awkwardly phrased: "subnormal numbers are the subset of denormalized numbers (sometimes called denormals)" would be better as "subnormal numbers (sometimes called denormals) are the subset of denormalized numbers". It does imply there are other "denormalized" numbers though; not sure what they are.
  - jcranmer 2y ago
    Terminology is confusing! A floating-point number is generally defined as (-1)^sign * (mantissa) * base^exponent, where mantissa is a sequence of digits in the base. A normalized floating-point number has mantissa be a fixed number of digits for a given floating-point type. A denormalized floating-point number is one with a different number of digits, where the exponent is the minimum possible value. An unnormalized floating-point number is one with a different number of digits, but the exponent is not the minimum possible value. (This doesn't occur in the standard IEEE 754 types, but the PPC and the x86 long double floating-point types have these numbers). There's also noncanonical floating-point numbers, which are different representations for the same value. These occur, for example, with IEEE 754 decimal FP, as well as many pre-IEEE 754 types (where instead of subnormals, you tend to get noncanonical representations of 0). There's also another kind of scenario where you have multiple values that are all still canonical--for example, 0.0 and 0.00 are different canonical decimal FP values in IEEE 754 (they have different exponents, and I'm not going to attempt to remember what they are off the top of my head), and of course -0 and +0 are distinct values of 0 that work differently.
    - Asraelite 2y ago
      You didn't cover what exactly "subnormal" means
      - jcranmer 2y ago
        In all practical terms, subnormal is the same as denormal (a non-normalized number with the smallest possible exponent). I think this is a situation where there's confusion over precision of terminology, so things have shifted to a new term which is consistently used. Subnormal is now the new preferred terminology for denormal, I think out of confusion as to whether or not unnormal numbers were also denormal (I have never heard the term so used, but I'm also not a great connoisseur of non-IEEE 754 floating-point). There's a similar attempt to shift use of the term "mantissa" to "significand", since a mantissa implies that the number is in the range [0, 1), whereas the usual implementation in a floating point type is to use the range [1, base).
    - 2y ago
- titzer 2y ago
  > FP hardware usually special cases them to some slow but correct silicon This is basically only true on Intel chips. Subnormals are not that slow on modern ARM or AMD hardware.
  - adgjlsfhk1 2y ago
    This is only mostly true unfortunately. If I recall my tests on M1 M1 has a performance penalty for subnormals, but more like a 10x than the 200x that it causes on Intel.
- a_e_k 2y ago
  > So, in some code it can make sense to "zap" subnormals, converting them to zero since they're almost zero anyway. Flush to zero (FTZ) is the usual term for this.

by mkl 2y ago

tialaramex 2y ago

Huh, that Wiki link taught me something I didn't know - I hadn't realised a different FP representation could have Denormals (a word I knew) which weren't Subnormal (a new word to me, more precisely defining what's different about these values). The Subnormals (presumably any Denormals regardless?) tend to be annoyingly slow, because the FP hardware usually special cases them to some slow but correct silicon not a fast path for the usual case. So, in some code it can make sense to "zap" subnormals, converting them to zero since they're almost zero anyway. Obviously in other code this is a terrible idea, so your domain knowledge comes into play when authoring such code. For example if we're using -1 .. +1 as the 0dB for our Karaoke software, a subnormal is clearly inaudible noise, but it might make our sophisticated reverb simulator (so you can pretend you're singing in a big auditorium) a thousand times slower than normal, so, zap those subnormals before the simulator and we hear the same results but don't get punished with miserable performance.

mkl 2y ago
Denormal and subnormal are synonyms as far as I know and can find. That first sentence is awkwardly phrased: "subnormal numbers are the subset of denormalized numbers (sometimes called denormals)" would be better as "subnormal numbers (sometimes called denormals) are the subset of denormalized numbers". It does imply there are other "denormalized" numbers though; not sure what they are.
- jcranmer 2y ago
  Terminology is confusing! A floating-point number is generally defined as (-1)^sign * (mantissa) * base^exponent, where mantissa is a sequence of digits in the base. A normalized floating-point number has mantissa be a fixed number of digits for a given floating-point type. A denormalized floating-point number is one with a different number of digits, where the exponent is the minimum possible value. An unnormalized floating-point number is one with a different number of digits, but the exponent is not the minimum possible value. (This doesn't occur in the standard IEEE 754 types, but the PPC and the x86 long double floating-point types have these numbers). There's also noncanonical floating-point numbers, which are different representations for the same value. These occur, for example, with IEEE 754 decimal FP, as well as many pre-IEEE 754 types (where instead of subnormals, you tend to get noncanonical representations of 0). There's also another kind of scenario where you have multiple values that are all still canonical--for example, 0.0 and 0.00 are different canonical decimal FP values in IEEE 754 (they have different exponents, and I'm not going to attempt to remember what they are off the top of my head), and of course -0 and +0 are distinct values of 0 that work differently.
  - Asraelite 2y ago
    You didn't cover what exactly "subnormal" means
    - jcranmer 2y ago
      In all practical terms, subnormal is the same as denormal (a non-normalized number with the smallest possible exponent). I think this is a situation where there's confusion over precision of terminology, so things have shifted to a new term which is consistently used. Subnormal is now the new preferred terminology for denormal, I think out of confusion as to whether or not unnormal numbers were also denormal (I have never heard the term so used, but I'm also not a great connoisseur of non-IEEE 754 floating-point). There's a similar attempt to shift use of the term "mantissa" to "significand", since a mantissa implies that the number is in the range [0, 1), whereas the usual implementation in a floating point type is to use the range [1, base).
  - 2y ago
titzer 2y ago
> FP hardware usually special cases them to some slow but correct silicon This is basically only true on Intel chips. Subnormals are not that slow on modern ARM or AMD hardware.
- adgjlsfhk1 2y ago
  This is only mostly true unfortunately. If I recall my tests on M1 M1 has a performance penalty for subnormals, but more like a 10x than the 200x that it causes on Intel.
a_e_k 2y ago
> So, in some code it can make sense to "zap" subnormals, converting them to zero since they're almost zero anyway. Flush to zero (FTZ) is the usual term for this.