4 ms·

I'm still trying to understand what a subnormal number is; IE, I'm looking for the TLDR so I know just enough to know if I'm using them and need to learn more.

by gwbas1c 14d ago

I'm still trying to understand what a subnormal number is; IE, I'm looking for the TLDR so I know just enough to know if I'm using them and need to learn more. Unfortunately, the Wikipedia article, while probably being accurate, doesn't give a clear and concise answer. IE, is 0.0001 a subnormal? Or is it 0.000000000000000000001?

ant6n 14d ago
Usually IEEE floats have an implied 1 in the front. So for the standard represented numbers, there's some minimum number 1.bbbbbb.. * 2^-N. This allows 1bit more precision than is actually stored. between any two numbers, there's basically the same epsilon difference, but from the smallest number to zero it's bigger. A subnormal number breaks that convention, it just becomes 0.bbbbb... * 2^-N. As the numbers get smaller, the relative difference between the numbers gets larger. That also means their precision is smaller than the normal floats.
account42 14d ago
Floating point numbers are usually interpreted as sign * 1.mantissa * 2 ^ exponent where sign, mantissa and exponent are fixed bit width integers. The 1. before the number is normally implicit because it would be a waste of a bit to encode it when you could just use a diferent exponent to represent such a number. However with this simple scheme the number zero and a relatively large gap around it cannot be represented (relatively large to the gap between the smallest and next smalles number that can be represented). So there is a special case where for the smallest encodeable exponent the mantissa must also specify that 1. or 0. prefix. Because its a special case it needs special handling that clever silicon engineers might think is unimportant enough to handle in microcode instead of dedicated silicon. x86 has a mode to assume that all such small numbers are actually equal to zero which can then be handle without microcode fallback. Technically its even a bit more complicated because x86 has two different float implementations and for at least SSE floats you can control the denormals-are-zero and flush-(denormals)-to-zero-(when writing) modes independently. GCC -ffast-math actual enables that mode for the entire main thread. AFAIK ARM NEON always works in that mode so the Gravion and Apple benchmarks might be unfair here undless you compare with DAZ and FTZ enabled on Intel. No idea if the AMD benchmarks might have used different modes. Because the flags are global per thread you can easily have unrelated loaded libraries messing the benchmark up.
- ack_complete 14d ago
  > AFAIK ARM NEON always works in that mode This was only true for ARMv7 NEON (32-bit). ARMv8 / AArch64 NEON is IEEE compliant.
Someone 14d ago
That depends on how you store it. Each number can be written in infinitely many ways, for example 12, 1.2E1, and 0.012E3 all are “twelve” In (binary) IEEE floats, the canonical way to write floats is significant × 2^exponent with 1 ≤ significant < 2. So, “twelve” gets stored as 1.5 × 2³ and not as, for example, 0.375 × 2⁵, 12 × 2⁰ or 96 × 2⁻³. Float operations normally return numbers satisfying that. However, in IEEE, the exponent cannot be made arbitrary small. Because of that, some very small numbers cannot be represented that way. In those cases the standard says operations can return numbers with the value closest to the correct value with a significant less than 1. Those number representations are called subnormals.
dgrunwald 14d ago
For 32-bit floats, subnormals are the numbers closer to 0 than 2**(-126) == 0.0000000000000000000000000000000000000117549. For 64-bit doubles, it's 2**(-1022), a number starting with 308 decimal zeroes.
- fuzzfactor 14d ago
  What about 8-bit and 16-bit?
  - nayuki 14d ago
    8-bit: There is no IEEE 754 standard format. 16-bit: https://en.wikipedia.org/wiki/Half-precision_floating-point_format https://en.wikipedia.org/wiki/Half-precision_floating-point_...
  - adrian_b 13d ago
    It should be noted that subnormals have appeared for the first time in standards a decade before Intel 8087 (the ancestor of the IEEE standard), in the standards for digital telephony, and that happened in a certain form of FP8. When telephony transitioned from analog voice transmission to digital, the PCM (pulse-code modulation) encoded audio signal used 8-bit samples, which were a form of 8-bit floating-point numbers (with American and European encoding variants: mu-law and A-law). The use of a floating-point format enabled the 8-bit samples to have a dynamic range as big as for a 12-bit or 13-bit fixed-point encoding. Subnormals where used in digital telephony, because otherwise the errors around zero would have been so great that the voice audio would not have been intelligible. In general, in smaller floating-point formats the use of subnormals is even more important than in bigger formats, in order to avoid the loss of precision around zero.
tialaramex 14d ago
The 32-bit subnormals are all the non-zero 32-bit floating point values between but not including -0.000000000000000000000000000000000000011754943508222875079687365372222456778186655567720875215087517062784172594547271728515625 and +0.000000000000000000000000000000000000011754943508222875079687365372222456778186655567720875215087517062784172594547271728515625 Does that help you? [Edited: correct decimal after noticing that my calculator defaulted to the wrong setting] [And again because I think there's a bug in the last few digits, so debugging that's a fun activity for the weekend] [And a third time because nope, those were correct and I can't type]
- gwbas1c 14d ago
  Yes, that's a perfect explanation! (And thankfully, for when I work with small numbers, they are still much larger than that.)
- nayuki 14d ago
  Very interesting, you are right. The IEEE 754 standard defines positive/negative zero as not subnormal numbers. Mechanically speaking, the two zeros use the subnormal number format, so in that sense they are subnormal (but definitionally they aren't). Also, I guess FPUs treat zero differently from other subnormal numbers, which is why zero doesn't have a performance penalty. Relevant articles to read: https://stackoverflow.com/questions/73890260/why-is-zero-not-a-subnormal-number https://stackoverflow.com/questions/73890260/why-is-zero-not... , https://en.wikipedia.org/wiki/Sterbenz_lemma https://en.wikipedia.org/wiki/Sterbenz_lemma , https://en.wikipedia.org/wiki/Subnormal_number https://en.wikipedia.org/wiki/Subnormal_number
jcranmer 14d ago
Binary floating-point numbers are scientific notation except the pieces are all in binary. In proper scientific notation, the only time the digit before the decimal point can be 0 is when the number itself is 0. Since the only other digit in binary notation is 1, there is no need to store the digit before the decimal point, since it's always 1... except now you can't store 0. This problem is fixed by reserving one of the exponents for the representation of 0. Some of the formats (e.g. VAX floating point) that introduced this implicit-1-bit for the binary format said that every number with this special-0-exponent was a zero. But IEEE 754 introduced the concept of gradual underflow, and says instead that it is a bit string with the implicit digit before the decimal point as a 0 instead of 1. Putting it differently and more succinctly: a subnormal number is a number that has fewer digits of precision than is normally implied by the format. Which numbers are subnormal numbers is entirely dependent on the floating-point format.

by gwbas1c 14d ago

ant6n 14d ago

Usually IEEE floats have an implied 1 in the front. So for the standard represented numbers, there's some minimum number 1.bbbbbb.. * 2^-N. This allows 1bit more precision than is actually stored. between any two numbers, there's basically the same epsilon difference, but from the smallest number to zero it's bigger. A subnormal number breaks that convention, it just becomes 0.bbbbb... * 2^-N. As the numbers get smaller, the relative difference between the numbers gets larger. That also means their precision is smaller than the normal floats.

account42 14d ago

Floating point numbers are usually interpreted as sign * 1.mantissa * 2 ^ exponent where sign, mantissa and exponent are fixed bit width integers. The 1. before the number is normally implicit because it would be a waste of a bit to encode it when you could just use a diferent exponent to represent such a number. However with this simple scheme the number zero and a relatively large gap around it cannot be represented (relatively large to the gap between the smallest and next smalles number that can be represented). So there is a special case where for the smallest encodeable exponent the mantissa must also specify that 1. or 0. prefix. Because its a special case it needs special handling that clever silicon engineers might think is unimportant enough to handle in microcode instead of dedicated silicon. x86 has a mode to assume that all such small numbers are actually equal to zero which can then be handle without microcode fallback. Technically its even a bit more complicated because x86 has two different float implementations and for at least SSE floats you can control the denormals-are-zero and flush-(denormals)-to-zero-(when writing) modes independently. GCC -ffast-math actual enables that mode for the entire main thread. AFAIK ARM NEON always works in that mode so the Gravion and Apple benchmarks might be unfair here undless you compare with DAZ and FTZ enabled on Intel. No idea if the AMD benchmarks might have used different modes. Because the flags are global per thread you can easily have unrelated loaded libraries messing the benchmark up.

ack_complete 14d ago
> AFAIK ARM NEON always works in that mode This was only true for ARMv7 NEON (32-bit). ARMv8 / AArch64 NEON is IEEE compliant.

Someone 14d ago

That depends on how you store it. Each number can be written in infinitely many ways, for example 12, 1.2E1, and 0.012E3 all are “twelve” In (binary) IEEE floats, the canonical way to write floats is significant × 2^exponent with 1 ≤ significant < 2. So, “twelve” gets stored as 1.5 × 2³ and not as, for example, 0.375 × 2⁵, 12 × 2⁰ or 96 × 2⁻³. Float operations normally return numbers satisfying that. However, in IEEE, the exponent cannot be made arbitrary small. Because of that, some very small numbers cannot be represented that way. In those cases the standard says operations can return numbers with the value closest to the correct value with a significant less than 1. Those number representations are called subnormals.

dgrunwald 14d ago

For 32-bit floats, subnormals are the numbers closer to 0 than 2**(-126) == 0.0000000000000000000000000000000000000117549. For 64-bit doubles, it's 2**(-1022), a number starting with 308 decimal zeroes.

fuzzfactor 14d ago
What about 8-bit and 16-bit?
- nayuki 14d ago
  8-bit: There is no IEEE 754 standard format. 16-bit: https://en.wikipedia.org/wiki/Half-precision_floating-point_format https://en.wikipedia.org/wiki/Half-precision_floating-point_...
- adrian_b 13d ago
  It should be noted that subnormals have appeared for the first time in standards a decade before Intel 8087 (the ancestor of the IEEE standard), in the standards for digital telephony, and that happened in a certain form of FP8. When telephony transitioned from analog voice transmission to digital, the PCM (pulse-code modulation) encoded audio signal used 8-bit samples, which were a form of 8-bit floating-point numbers (with American and European encoding variants: mu-law and A-law). The use of a floating-point format enabled the 8-bit samples to have a dynamic range as big as for a 12-bit or 13-bit fixed-point encoding. Subnormals where used in digital telephony, because otherwise the errors around zero would have been so great that the voice audio would not have been intelligible. In general, in smaller floating-point formats the use of subnormals is even more important than in bigger formats, in order to avoid the loss of precision around zero.

tialaramex 14d ago

The 32-bit subnormals are all the non-zero 32-bit floating point values between but not including -0.000000000000000000000000000000000000011754943508222875079687365372222456778186655567720875215087517062784172594547271728515625 and +0.000000000000000000000000000000000000011754943508222875079687365372222456778186655567720875215087517062784172594547271728515625 Does that help you? [Edited: correct decimal after noticing that my calculator defaulted to the wrong setting] [And again because I think there's a bug in the last few digits, so debugging that's a fun activity for the weekend] [And a third time because nope, those were correct and I can't type]

gwbas1c 14d ago
Yes, that's a perfect explanation! (And thankfully, for when I work with small numbers, they are still much larger than that.)
nayuki 14d ago
Very interesting, you are right. The IEEE 754 standard defines positive/negative zero as not subnormal numbers. Mechanically speaking, the two zeros use the subnormal number format, so in that sense they are subnormal (but definitionally they aren't). Also, I guess FPUs treat zero differently from other subnormal numbers, which is why zero doesn't have a performance penalty. Relevant articles to read: https://stackoverflow.com/questions/73890260/why-is-zero-not-a-subnormal-number https://stackoverflow.com/questions/73890260/why-is-zero-not... , https://en.wikipedia.org/wiki/Sterbenz_lemma https://en.wikipedia.org/wiki/Sterbenz_lemma , https://en.wikipedia.org/wiki/Subnormal_number https://en.wikipedia.org/wiki/Subnormal_number

jcranmer 14d ago

Binary floating-point numbers are scientific notation except the pieces are all in binary. In proper scientific notation, the only time the digit before the decimal point can be 0 is when the number itself is 0. Since the only other digit in binary notation is 1, there is no need to store the digit before the decimal point, since it's always 1... except now you can't store 0. This problem is fixed by reserving one of the exponents for the representation of 0. Some of the formats (e.g. VAX floating point) that introduced this implicit-1-bit for the binary format said that every number with this special-0-exponent was a zero. But IEEE 754 introduced the concept of gradual underflow, and says instead that it is a bit string with the implicit digit before the decimal point as a 0 instead of 1. Putting it differently and more succinctly: a subnormal number is a number that has fewer digits of precision than is normally implied by the format. Which numbers are subnormal numbers is entirely dependent on the floating-point format.