4 ms·

what does subnormal mean?

camel-cdr 3y ago
Usually floating point numbers are defined as `1.fraction * 2^(exponent-bias)`, but if the exponent is zero `0.fraction * 2^(1-bias)` is used, to allow representing very small numbers. IIRC I also explained it here: https://m.youtube.com/watch?v=VHJUlRiRDCY&t=2774s https://m.youtube.com/watch?v=VHJUlRiRDCY&t=2774s
- NeutralForest 3y ago
  thanks!
- Aardwolf 3y ago
  And the reason subnormals exist at all is that normally there's an implicit 1-bit (without the implicit 1-bit, there would be multiple representations of the same value, e.g. all values with mantissa 0 would represent 0 regardless of exponent), but you could never represent the value 0 itself if the implicit 1-bit is always there. So for the lowest exponent value, the implicit bit is set to 0, to allow representing 0, but also those other very tiny values since there's still an entire mantissa to give meaning to in addition to value 0.
  - magicalhippo 3y ago
    Subnormals or denormalized floats can lead to some unexpected results. IIRC dividing a subnormal by a subnormal returns infinity. I experienced this where a colleague had tried to reduce the number of divisions by rewriting (a/x)*(b/y)*(c/z) to (a*b*c)/(x*y*z). The problem was that in certain circumstances the terms were all small, but of comparable magnitude. Thus the latter lead to subnormal divided by subnormal. The fix was to undo the optimization and do the divisions first, which lead to the multiplication of three numbers of order 1.
    - bee_rider 3y ago
      Yeah, I think a good general guideline is that if you see a bunch of divisions like that, the author did it intentionally (to avoid overflow or subnormals). Anyway, I wonder if the optimization even helps at all. Either way the critical path is a division and two multiplications. That seems like the sort of thing that, between SIMD an OoO execution, a modern CPU ought to be able to work out, haha. No promises but I’d want to benchmark it before caring.
    - Aardwolf 3y ago
      Maybe depends on the platform? I tried 5e-323 / 6e-323 in javascript console and it returns 0.8333333333333334 (and e-323 should be subnormal for double precision)
      - leiroigh 3y ago
        Depends on control registers like e.g. MXCSR. It's an utter mess. Consider e.g. https://news.ycombinator.com/item?id=32738206 https://news.ycombinator.com/item?id=32738206
    - thethirdone 3y ago
      > IIRC dividing a subnormal by a subnormal returns infinity. This should not be true in a conformant implementation of IEEE 754-2008. You can get infinity by dividing by a subnormal (due to overflow), but that should only be possible with sufficiently large numerators. > The problem was that in certain circumstances the terms were all small, but of comparable magnitude. Thus the latter lead to subnormal divided by subnormal. The fix was to undo the optimization and do the divisions first, which lead to the multiplication of three numbers of order 1. My guess is that this problem was actually the numerator being subnormal and the denominator underflowing all the way to zero and therefore the result going to infinity.
      - magicalhippo 3y ago
        It's been over a decade, so I struggle to recall the details. > My guess is that this problem was actually the numerator being subnormal and the denominator underflowing all the way to zero and therefore the result going to infinity. Yes thinking about you're probably correct.

Usually floating point numbers are defined as `1.fraction * 2^(exponent-bias)`, but if the exponent is zero `0.fraction * 2^(1-bias)` is used, to allow representing very small numbers. IIRC I also explained it here: https://m.youtube.com/watch?v=VHJUlRiRDCY&t=2774s https://m.youtube.com/watch?v=VHJUlRiRDCY&t=2774s

NeutralForest 3y ago
thanks!
Aardwolf 3y ago
And the reason subnormals exist at all is that normally there's an implicit 1-bit (without the implicit 1-bit, there would be multiple representations of the same value, e.g. all values with mantissa 0 would represent 0 regardless of exponent), but you could never represent the value 0 itself if the implicit 1-bit is always there. So for the lowest exponent value, the implicit bit is set to 0, to allow representing 0, but also those other very tiny values since there's still an entire mantissa to give meaning to in addition to value 0.
- magicalhippo 3y ago
  Subnormals or denormalized floats can lead to some unexpected results. IIRC dividing a subnormal by a subnormal returns infinity. I experienced this where a colleague had tried to reduce the number of divisions by rewriting (a/x)*(b/y)*(c/z) to (a*b*c)/(x*y*z). The problem was that in certain circumstances the terms were all small, but of comparable magnitude. Thus the latter lead to subnormal divided by subnormal. The fix was to undo the optimization and do the divisions first, which lead to the multiplication of three numbers of order 1.
  - bee_rider 3y ago
    Yeah, I think a good general guideline is that if you see a bunch of divisions like that, the author did it intentionally (to avoid overflow or subnormals). Anyway, I wonder if the optimization even helps at all. Either way the critical path is a division and two multiplications. That seems like the sort of thing that, between SIMD an OoO execution, a modern CPU ought to be able to work out, haha. No promises but I’d want to benchmark it before caring.
  - Aardwolf 3y ago
    Maybe depends on the platform? I tried 5e-323 / 6e-323 in javascript console and it returns 0.8333333333333334 (and e-323 should be subnormal for double precision)
    - leiroigh 3y ago
      Depends on control registers like e.g. MXCSR. It's an utter mess. Consider e.g. https://news.ycombinator.com/item?id=32738206 https://news.ycombinator.com/item?id=32738206
  - thethirdone 3y ago
    > IIRC dividing a subnormal by a subnormal returns infinity. This should not be true in a conformant implementation of IEEE 754-2008. You can get infinity by dividing by a subnormal (due to overflow), but that should only be possible with sufficiently large numerators. > The problem was that in certain circumstances the terms were all small, but of comparable magnitude. Thus the latter lead to subnormal divided by subnormal. The fix was to undo the optimization and do the divisions first, which lead to the multiplication of three numbers of order 1. My guess is that this problem was actually the numerator being subnormal and the denominator underflowing all the way to zero and therefore the result going to infinity.
    - magicalhippo 3y ago
      It's been over a decade, so I struggle to recall the details. > My guess is that this problem was actually the numerator being subnormal and the denominator underflowing all the way to zero and therefore the result going to infinity. Yes thinking about you're probably correct.