4 ms·

The Secret Life of NaN (2018)

52 points by prakashqwerty 5mo ago

WalterBright 5mo ago
NaNs are a very underappreciated feature of IEEE-754 floating point. In the D programming language, floats get default initialized to NaN, not to 0.0. double y = 0.0; // initialized to 0.0 double x; // initialized to NaN The discussion routinely comes up as "why not default initialize to 0.0?" The reason is a routine mistake in programming is forgetting to initialize a variable. With a floating point 0.0, one may never realize that the floating point calculation results are wrong. But with NaN, the result of a floating point computation will be NaN, which is unlikely to go unnoticed. I don't know of any other programming language with this safety feature. Also, the D `char` type is initialized to 0xFF, not 0, because Unicode says that 0xFF is an invalid character.
- WalterBright 5mo ago
  Another crucial use of NaNs is if you have a sensor. If the sensor has failed, the sensed value should be transmitted as NaN, not 0, so the receiver knows the data is bad.
  - AlotOfReading 5mo ago
    My experience is that if you write an interface that (rarely) returns NaNs, someone will use it assuming it's never NaN no matter how good the docs are. Then their code does bad things and you have to patiently explain why they're wrong and yes, they are holding isnan() wrong (in C/C++).
    - WalterBright 5mo ago
      NaN for a failed sensor is objectively better than any other value. But at some point you just cannot help some people.
    - adrian_b 5mo ago
      When such users are expected, there exists only one solution. Do not mask the invalid operation exception, which was actually the original recommendation of the IEEE standard, which was that the default behavior should be to mask all exceptions, except the invalid operation exception. When the invalid operation exception is not masked, NaNs are never generated and any NaN present in the input data will generate an exception, which will abort the program, unless the exception is handled. This behavior avoids the bugs caused by careless programmers. Unfortunately, the original suggestion was not adopted by most programming language implementers, so nowadays the typical default setting is to have all exceptions masked. When the programmers also omit to handle the special values, bugs may remain unnoticed. Special values need not be handled everywhere, because infinities and NaNs will propagate through many operations, so they will remain in the final results. But wherever a value is not persistent, but it is used in some decision and it is discarded after that, special values like NaNs must be handled correctly.
    - deleted 5mo ago
      [deleted]
  - bumby 5mo ago
    Doesn’t this completely depend on the sensor failure mode? Eg if a voltage sensor internally shorts to ground, the failure will read 0V, not NaN. Or are you using “failed sensor” to only mean “not reporting” here? I think your initialization is smart in many use cases, but the sensor application probably isn’t one of them except for that single failure mode. It can still lead to masked failures and false assumptions (“the sensor is getting a value so it must be working”). That’s the same issue as what you’re supposedly fixing by that design choice. It still requires engineering knowledge to assess correctly.
    - WalterBright 5mo ago
      Yes, I assume the sensor is designed to detect its own failures. If a sensor is capable of emitting floating point values, surely its software can emit a NaN. The point of a NaN value is it does not require sophisticated engineering knowledge to realize that a NaN output is not what you're expecting.
      - bumby 5mo ago
        >I assume the sensor is designed to detect its own failures. Bold assumption. I would be willing to bet this is more the exception than the rule on most sensors/systems. >The point of a NaN value is it does not require sophisticated engineering knowledge to realize that a NaN output is not what you're expecting. What I was pointing out is this only captures a relatively narrow set of failure modes and may lead to bad assumptions due to automation bias. E.g., "I only need to think about failures if the sensor gives an NaN because it's based on the assumption that a failure produces an NaN" whereas having an actual principled knowledge of operation can catch the other errors.
- p1necone 5mo ago
  Just requiring explicit assignment before first use feels like the superior approach to automatic initialization, regardless of whether the automatic initialization is with 0 or with NaN.
  - WalterBright 5mo ago
    That suggestion is often made. The trouble with it is a bug I've seen often. People will get an error message about an "uninitialized variable". Then they go into "just get the compiler to shut up" mode, amd pick "0" as the initializer. Then, the program compiles and runs, and silently produces the wrong answer. Code reviews will simply pass over the "0" initializer, as it looks right. With default NaN initialization, the programmer is more likely to stop and think about it, not just insert 0. Another issue with it is: float x = 0.0; setFloat(&x); void setFloat(float* px) { *px = 3.0; } For the purposes of code clarity I don't want to see a variable initialized to a value that is never used, just to shut the compiler up.
    - ncurses1010 5mo ago
      With the default initialization to nan, do you ever run into situations where people are searching for common sources for nan (nan literals, div by zero) and they can't find it? Or cases where only some branches but not others initialize the float?
      - WalterBright 5mo ago
        To leave a variable uninitialized, use the construction: int x = void; Note that nobody is going to write this by accident. And it's easy to grep for. To find the source of a NaN, it helps to know that every operation that has a NaN as an operand produces a NaN as a result. So if you see a NaN in the output, you can work backwards to where it originated.
        jcranmer 5mo ago
        > every operation that has a NaN as an operand produces a NaN as a result. That's not true. The minimum/maximum functions (fmin and fminimum_num variants, but not the fminimum one) treat NaN inputs as not-present, so return the non-NaN value if there is one. Similarly, hypot also treats NaN inputs as not-present. pow and compoundn will ignore NaN exponents if the base is 1.
- wpollock 5mo ago
  > ... Unicode says that 0xFF is an invalid character. Not so. You may be thinking of UTF-8 encoding. 0xff is DEL in Unicode.
  - WalterBright 5mo ago
    The "char" type in D represents a UTF-8 code unit, the byte 0xFF is not a valid character code and is strictly forbidden.
  - LittleLily 5mo ago
    DEL is unicode codepoint U+007F, which is the byte 0x7F in UTF-8, not 0xFF. Perhaps you were thinking of ÿ which is codepoint U+00FF, which encodes to the bytes 0xC3 0xBF in UTF-8.
    - wpollock 5mo ago
      I was thinking of DEL, but was obviously mistaken. Thanks for catching that!
- anitil 5mo ago
  That's a very thoughtful decision, I always enjoy your updates on D
- addaon 5mo ago
  What's the cost of this in terms of not being able to bzero() simple data structures, or use OS-cleared pages directly without dirtying them? This seems like it would turn some sparse memory usage patterns dense…
  - WalterBright 5mo ago
    You can always statically initialize them with 0: static float[10] array = 0.0;
GMoromisato 5mo ago
I use nan boxing in GridWhale. It feels like the Infinite Hotel[1]: you can always add another type. Note that these techniques also rely on the fact that we don't use all 64-bits for memory addressing. If we ever do, lots of VMs will break. For me, the major advantage of nan boxing is that you don't have to allocate a whole class of types (like floats). That saves so much at garbage collection time. ------------ [1] https://en.wikipedia.org/wiki/Hilbert%27s_paradox_of_the_Grand_Hotel https://en.wikipedia.org/wiki/Hilbert%27s_paradox_of_the_Gra...
jasperry 5mo ago
This is super useful, thanks. So if I were implementing a programming language, and wanted to have symbols to specify NaN in source code, I'd really only need quiet NaN, right? Because signaling NaN is supposed to always to raise an exception anyway?
- WalterBright 5mo ago
  I originally implemented Signalling and Quiet NaNs in the compiler. It was an abject failure. With all the transformations a compiler does, where the signalling turns into a quiet is lost. So just quiet NaNs are used.

52 points by prakashqwerty 5mo ago

NaNs are a very underappreciated feature of IEEE-754 floating point. In the D programming language, floats get default initialized to NaN, not to 0.0. double y = 0.0; // initialized to 0.0 double x; // initialized to NaN The discussion routinely comes up as "why not default initialize to 0.0?" The reason is a routine mistake in programming is forgetting to initialize a variable. With a floating point 0.0, one may never realize that the floating point calculation results are wrong. But with NaN, the result of a floating point computation will be NaN, which is unlikely to go unnoticed. I don't know of any other programming language with this safety feature. Also, the D `char` type is initialized to 0xFF, not 0, because Unicode says that 0xFF is an invalid character.

WalterBright 5mo ago
Another crucial use of NaNs is if you have a sensor. If the sensor has failed, the sensed value should be transmitted as NaN, not 0, so the receiver knows the data is bad.
- AlotOfReading 5mo ago
  My experience is that if you write an interface that (rarely) returns NaNs, someone will use it assuming it's never NaN no matter how good the docs are. Then their code does bad things and you have to patiently explain why they're wrong and yes, they are holding isnan() wrong (in C/C++).
  - WalterBright 5mo ago
    NaN for a failed sensor is objectively better than any other value. But at some point you just cannot help some people.
  - adrian_b 5mo ago
    When such users are expected, there exists only one solution. Do not mask the invalid operation exception, which was actually the original recommendation of the IEEE standard, which was that the default behavior should be to mask all exceptions, except the invalid operation exception. When the invalid operation exception is not masked, NaNs are never generated and any NaN present in the input data will generate an exception, which will abort the program, unless the exception is handled. This behavior avoids the bugs caused by careless programmers. Unfortunately, the original suggestion was not adopted by most programming language implementers, so nowadays the typical default setting is to have all exceptions masked. When the programmers also omit to handle the special values, bugs may remain unnoticed. Special values need not be handled everywhere, because infinities and NaNs will propagate through many operations, so they will remain in the final results. But wherever a value is not persistent, but it is used in some decision and it is discarded after that, special values like NaNs must be handled correctly.
  - deleted 5mo ago
    [deleted]
- bumby 5mo ago
  Doesn’t this completely depend on the sensor failure mode? Eg if a voltage sensor internally shorts to ground, the failure will read 0V, not NaN. Or are you using “failed sensor” to only mean “not reporting” here? I think your initialization is smart in many use cases, but the sensor application probably isn’t one of them except for that single failure mode. It can still lead to masked failures and false assumptions (“the sensor is getting a value so it must be working”). That’s the same issue as what you’re supposedly fixing by that design choice. It still requires engineering knowledge to assess correctly.
  - WalterBright 5mo ago
    Yes, I assume the sensor is designed to detect its own failures. If a sensor is capable of emitting floating point values, surely its software can emit a NaN. The point of a NaN value is it does not require sophisticated engineering knowledge to realize that a NaN output is not what you're expecting.
    - bumby 5mo ago
      >I assume the sensor is designed to detect its own failures. Bold assumption. I would be willing to bet this is more the exception than the rule on most sensors/systems. >The point of a NaN value is it does not require sophisticated engineering knowledge to realize that a NaN output is not what you're expecting. What I was pointing out is this only captures a relatively narrow set of failure modes and may lead to bad assumptions due to automation bias. E.g., "I only need to think about failures if the sensor gives an NaN because it's based on the assumption that a failure produces an NaN" whereas having an actual principled knowledge of operation can catch the other errors.
p1necone 5mo ago
Just requiring explicit assignment before first use feels like the superior approach to automatic initialization, regardless of whether the automatic initialization is with 0 or with NaN.
- WalterBright 5mo ago
  That suggestion is often made. The trouble with it is a bug I've seen often. People will get an error message about an "uninitialized variable". Then they go into "just get the compiler to shut up" mode, amd pick "0" as the initializer. Then, the program compiles and runs, and silently produces the wrong answer. Code reviews will simply pass over the "0" initializer, as it looks right. With default NaN initialization, the programmer is more likely to stop and think about it, not just insert 0. Another issue with it is: float x = 0.0; setFloat(&x); void setFloat(float* px) { *px = 3.0; } For the purposes of code clarity I don't want to see a variable initialized to a value that is never used, just to shut the compiler up.
  - ncurses1010 5mo ago
    With the default initialization to nan, do you ever run into situations where people are searching for common sources for nan (nan literals, div by zero) and they can't find it? Or cases where only some branches but not others initialize the float?
    - WalterBright 5mo ago
      To leave a variable uninitialized, use the construction: int x = void; Note that nobody is going to write this by accident. And it's easy to grep for. To find the source of a NaN, it helps to know that every operation that has a NaN as an operand produces a NaN as a result. So if you see a NaN in the output, you can work backwards to where it originated.
      - jcranmer 5mo ago
        > every operation that has a NaN as an operand produces a NaN as a result. That's not true. The minimum/maximum functions (fmin and fminimum_num variants, but not the fminimum one) treat NaN inputs as not-present, so return the non-NaN value if there is one. Similarly, hypot also treats NaN inputs as not-present. pow and compoundn will ignore NaN exponents if the base is 1.
wpollock 5mo ago
> ... Unicode says that 0xFF is an invalid character. Not so. You may be thinking of UTF-8 encoding. 0xff is DEL in Unicode.
- WalterBright 5mo ago
  The "char" type in D represents a UTF-8 code unit, the byte 0xFF is not a valid character code and is strictly forbidden.
- LittleLily 5mo ago
  DEL is unicode codepoint U+007F, which is the byte 0x7F in UTF-8, not 0xFF. Perhaps you were thinking of ÿ which is codepoint U+00FF, which encodes to the bytes 0xC3 0xBF in UTF-8.
  - wpollock 5mo ago
    I was thinking of DEL, but was obviously mistaken. Thanks for catching that!
anitil 5mo ago
That's a very thoughtful decision, I always enjoy your updates on D
addaon 5mo ago
What's the cost of this in terms of not being able to bzero() simple data structures, or use OS-cleared pages directly without dirtying them? This seems like it would turn some sparse memory usage patterns dense…
- WalterBright 5mo ago
  You can always statically initialize them with 0: static float[10] array = 0.0;

GMoromisato 5mo ago

I use nan boxing in GridWhale. It feels like the Infinite Hotel[1]: you can always add another type. Note that these techniques also rely on the fact that we don't use all 64-bits for memory addressing. If we ever do, lots of VMs will break. For me, the major advantage of nan boxing is that you don't have to allocate a whole class of types (like floats). That saves so much at garbage collection time. ------------ [1] https://en.wikipedia.org/wiki/Hilbert%27s_paradox_of_the_Grand_Hotel https://en.wikipedia.org/wiki/Hilbert%27s_paradox_of_the_Gra...

jasperry 5mo ago

This is super useful, thanks. So if I were implementing a programming language, and wanted to have symbols to specify NaN in source code, I'd really only need quiet NaN, right? Because signaling NaN is supposed to always to raise an exception anyway?

WalterBright 5mo ago
I originally implemented Signalling and Quiet NaNs in the compiler. It was an abject failure. With all the transformations a compiler does, where the signalling turns into a quiet is lost. So just quiet NaNs are used.