5 ms·

Does this correctly handle all the different NaN values, weird corner cases, etc. properly?

JadeNB 5y ago
> Does this correctly handle all the different NaN values, weird corner cases, etc. properly? Of course the answer to "have you covered all weird corner cases" is always 'no', but the article addresses the issue of different NaNs: > […] However, it turns out that doubleToLongBits is also not entirely trivial, because it canonicalizes NaNs. There are many ways to encode a not-a-number as a double, but only one of them is canonical. These different NaNs are even more similar twins, they can be distinguished neither via Double.compare, nor via any operation. Even string representation is the same. But they look different in computer memory. To avoid surprises, doubleToLongBits converts any NaN to the canonical form, which is encoded in long as 0x7ff8000000000000L. Of course, this procedure adds more conditions, which we do not need here. > What can we do? It appears that there's another method doubleToRawLongBits. It doesn't do any smart conversion of NaN and just returns exactly the same bit representation: […]
- sp332 5y ago
  The question is whether that last technique, of just flipping the sign bit, handles NaNs and infinities.
  - MontyCarloHall 5y ago
    Yes. The sign bit is totally independent of the value being encoded, whether that’s a number, q/sNaN, or infinity. Nitpick: we are not flipping the sign bit, but rather setting it to 0.
  - pm215 5y ago
    It's the required semantics of the IEEE floating point spec's absolute-value operation. This does mean that abs() doesn't behave like eg addition in that it won't set fp exception flags for signaling NaNs, or canonicalize NaNs, or set a flag to note that this was a denormal number. But at the language level "follows IEEE semantics" is probably overall less surprising. chs() ("flip the sign bit") is the other operation that's defined as effectively just an operation on the representation rather than a "do a real floating point computation".
  - klyrs 5y ago
    Yeah, the title is a head-scratcher. It is simple to calculate the absolute value. Only, it doesn't look simple in a high-level language.
pedrocr 5y ago
For double it would be harder to check but for float (that should have the same corner cases) it's easy to just check all possible values if you have a gold standard. I've been doing that for image processing functions and it's amazing how many corner cases you fix and keep fixed by writing unit tests that iterate over all possible values or a good subset of them when that's not possible.
- Strilanc 5y ago
  I second "just try all of them" when testing single precision floats. It's really easy to do, and gives you a lot of confidence in the correctness. For example, I had a float-to-bytes-to-floats conversion in some GPU shaders and "just test that every single possible float survives a round trip" made it trivial to be sure it was correct. (Also I found out my video card drivers were losing an ulp when parsing float constants in shaders.)
  - pedrocr 5y ago
    Roundtrip tests are great when you don't have a gold standard. It allows for at least consistency across operations. I've done a bunch and found quite a few corner cases. I also did a few roundtrip tests in RGB spaces where being exhaustive was no longer feasible. For those I ended up iterating over the whole RGB space but skipping values in each component by different prime values, to try and cover as many of the possible weird cases as possible: https://github.com/pedrocr/imagepipe/blob/12269f04ac2fa0d165fc860d853ca828a8b8de3e/src/color_conversions.rs#L337-L610 https://github.com/pedrocr/imagepipe/blob/12269f04ac2fa0d165...

> Does this correctly handle all the different NaN values, weird corner cases, etc. properly? Of course the answer to "have you covered all weird corner cases" is always 'no', but the article addresses the issue of different NaNs: > […] However, it turns out that doubleToLongBits is also not entirely trivial, because it canonicalizes NaNs. There are many ways to encode a not-a-number as a double, but only one of them is canonical. These different NaNs are even more similar twins, they can be distinguished neither via Double.compare, nor via any operation. Even string representation is the same. But they look different in computer memory. To avoid surprises, doubleToLongBits converts any NaN to the canonical form, which is encoded in long as 0x7ff8000000000000L. Of course, this procedure adds more conditions, which we do not need here. > What can we do? It appears that there's another method doubleToRawLongBits. It doesn't do any smart conversion of NaN and just returns exactly the same bit representation: […]

sp332 5y ago
The question is whether that last technique, of just flipping the sign bit, handles NaNs and infinities.
- MontyCarloHall 5y ago
  Yes. The sign bit is totally independent of the value being encoded, whether that’s a number, q/sNaN, or infinity. Nitpick: we are not flipping the sign bit, but rather setting it to 0.
- pm215 5y ago
  It's the required semantics of the IEEE floating point spec's absolute-value operation. This does mean that abs() doesn't behave like eg addition in that it won't set fp exception flags for signaling NaNs, or canonicalize NaNs, or set a flag to note that this was a denormal number. But at the language level "follows IEEE semantics" is probably overall less surprising. chs() ("flip the sign bit") is the other operation that's defined as effectively just an operation on the representation rather than a "do a real floating point computation".
- klyrs 5y ago
  Yeah, the title is a head-scratcher. It is simple to calculate the absolute value. Only, it doesn't look simple in a high-level language.

pedrocr 5y ago

For double it would be harder to check but for float (that should have the same corner cases) it's easy to just check all possible values if you have a gold standard. I've been doing that for image processing functions and it's amazing how many corner cases you fix and keep fixed by writing unit tests that iterate over all possible values or a good subset of them when that's not possible.

Strilanc 5y ago
I second "just try all of them" when testing single precision floats. It's really easy to do, and gives you a lot of confidence in the correctness. For example, I had a float-to-bytes-to-floats conversion in some GPU shaders and "just test that every single possible float survives a round trip" made it trivial to be sure it was correct. (Also I found out my video card drivers were losing an ulp when parsing float constants in shaders.)
- pedrocr 5y ago
  Roundtrip tests are great when you don't have a gold standard. It allows for at least consistency across operations. I've done a bunch and found quite a few corner cases. I also did a few roundtrip tests in RGB spaces where being exhaustive was no longer feasible. For those I ended up iterating over the whole RGB space but skipping values in each component by different prime values, to try and cover as many of the possible weird cases as possible: https://github.com/pedrocr/imagepipe/blob/12269f04ac2fa0d165fc860d853ca828a8b8de3e/src/color_conversions.rs#L337-L610 https://github.com/pedrocr/imagepipe/blob/12269f04ac2fa0d165...