5 ms·

There's been a few articles about rounding floating point numbers correctly in the past few days, but I realized that might not always be what you want. For in

There's been a few articles about rounding floating point numbers correctly in the past few days, but I realized that might not always be what you want. For instance when adding some small positive number to a floating point value I would typically prefer it round up if correct rounding would mean that it remains unchanged. b ≠ 0 and a + b = a can be a source of fun bugs. However a point can be made that if you even need to consider such behavior, you should probably be using something more precise than your current number representation.

jimktrains2 5y ago
> However a point can be made that if you even need to consider such behavior, you should probably be using something more precise than your current number representation. Or a different algorithm. For instance, when computing a mean, keeping the current mean and count instead of the current total and count is more numerically stable. https://statisticsblog412536603.wordpress.com/2019/10/17/streaming-algorithms-for-the-mean-mu-and-the-variance-knouth-welford-sigma2-show-with-and-example-why-the-running-mean-formula-is-preferable-to-the-definition-formula/ https://statisticsblog412536603.wordpress.com/2019/10/17/str...
aardvark179 5y ago
If you require a distinct point then you can check if a is smaller than the minimum increment that can be added to b, and add that increment instead, but in the few cases I’ve needed that sort of behaviour it has been easier to do the calculation and then check for equality.
gliptic 5y ago
What about stochastic rounding as used in many quantized neural network implementations :).
- ChrisLomont 5y ago
  Leads to non-reproducible or non-portable software. It's better to have well-defined and reproducible behavior, so one can reason how to make correct algorithms. Neural networks do it for very precise reasons nearly unique to them. If they were built on randomly rounding primitives, and that random was not the precise flavor one model or another needed, then you'd not easily be able to get the distributions needed for the task.
deepsun 5y ago
Well, you shouldn't do exact equality check on floats anyway, that can be a source of fun bugs regardless of rounding.
- gugagore 5y ago
  I see this advice often and I find that it misses the point. The problem isn't the equality check, the problem is finite precision. There are situations that call for exact equality checking in floating point.
  - cratermoon 5y ago
    In which case, you don't use IEEE 754 floating point for math, you use some kind of numerics type or library that correctly models the domain.
mbauman 5y ago
That's just trading one class of bugs for another. Adding a million 10^-50s to 1.0 is far still closer to 1.0 than the next floating point number. Your algorithm would give an answer that's a million values away.
- chmod775 5y ago
  It's a trade off regardless of what you're doing. One class of bug can make you overwrite values in a map or get your code 'stuck' to a number, the other just gives you a (more) wrong number. I know which kind of error would be preferred in a video game for instance.
  - mbauman 5y ago
    > the other just gives you a (more) wrong number. This is a common misconception: IEEE754 floating point arithmetic is not wrong or fuzzy or inherently inexact. It is very exact; it just has limited precision.
    - chmod775 5y ago
      No. In mathematical terms typical floating point arithmetic has perfect precision - it is repeatable - but limited accuracy - it can deviate from the real value. It's a shortcoming of CS vocabulary that "precision" is also used to refer to the number of bits used. "Exactness" is a fuzzy term used in common english to refer to the concept of both precision and accuracy, or either really. It's not really used in mathematics.
      - mbauman 5y ago
        Oh cool, you want to use different definitions. No matter. Point remains: the floating point representation of 1/4 is not approximate. Similarly, the closest floating point value to 1/10 also precisely defines some value: it's exactly 0.1000000000000000055511151231257827021181583404541015625. Now whether that's accurate to you depends upon your application. But the numbers don't care.
da_chicken 5y ago
> However a point can be made that if you even need to consider such behavior, you should probably be using something more precise than your current number representation. I would agree with this. If you're in a place where you actually need to care about how accurately you're representing numbers with IEEE 754, you almost certainly should look at a precise numeric data type instead and should set aside IEEE 754 entirely. It's less common that we think that we need both the range of values that a float or double represent and need those values represented precisely enough to endure aggregation. It would be nice to be able to give fewer digits to the exponent in a double, though. How often have I really needed 1e308 or 1e-324? How often have I even needed 1e38 or 1e-45? Then again, I don't remember how much precision we'd get back with more digits in the mantissa. Precision is a really funny thing, too. Remember, pi to ~37 decimal places is precise enough to calculate the circumference of a circle of atoms (~10^-10 m) the size of the observable universe (~10^26 m), as at that point the primary source of imprecision is the fact that you used atoms to construct your circle.
- tomsmeding 5y ago
  > It would be nice to be able to give fewer digits to the exponent in a double, though. How often have I really needed 1e308 or 1e-324? How often have I even needed 1e38 or 1e-45? Then again, I don't remember how much precision we'd get back with more digits in the mantissa. I'm not sure that helps. Going from 10^308 to 10^38, approximately, takes 3 bits from the 11-bit exponent and adds 3 bits to the 52-bit mantissa. This gives you _almost_ one extra decimal digit in your precision. It's not worth it, and if it is you certainly need a more special-purpose number type.

jimktrains2 5y ago

> However a point can be made that if you even need to consider such behavior, you should probably be using something more precise than your current number representation. Or a different algorithm. For instance, when computing a mean, keeping the current mean and count instead of the current total and count is more numerically stable. https://statisticsblog412536603.wordpress.com/2019/10/17/streaming-algorithms-for-the-mean-mu-and-the-variance-knouth-welford-sigma2-show-with-and-example-why-the-running-mean-formula-is-preferable-to-the-definition-formula/ https://statisticsblog412536603.wordpress.com/2019/10/17/str...

aardvark179 5y ago

If you require a distinct point then you can check if a is smaller than the minimum increment that can be added to b, and add that increment instead, but in the few cases I’ve needed that sort of behaviour it has been easier to do the calculation and then check for equality.

gliptic 5y ago

What about stochastic rounding as used in many quantized neural network implementations :).

ChrisLomont 5y ago
Leads to non-reproducible or non-portable software. It's better to have well-defined and reproducible behavior, so one can reason how to make correct algorithms. Neural networks do it for very precise reasons nearly unique to them. If they were built on randomly rounding primitives, and that random was not the precise flavor one model or another needed, then you'd not easily be able to get the distributions needed for the task.

deepsun 5y ago

Well, you shouldn't do exact equality check on floats anyway, that can be a source of fun bugs regardless of rounding.

gugagore 5y ago
I see this advice often and I find that it misses the point. The problem isn't the equality check, the problem is finite precision. There are situations that call for exact equality checking in floating point.
- cratermoon 5y ago
  In which case, you don't use IEEE 754 floating point for math, you use some kind of numerics type or library that correctly models the domain.

mbauman 5y ago

That's just trading one class of bugs for another. Adding a million 10^-50s to 1.0 is far still closer to 1.0 than the next floating point number. Your algorithm would give an answer that's a million values away.

chmod775 5y ago
It's a trade off regardless of what you're doing. One class of bug can make you overwrite values in a map or get your code 'stuck' to a number, the other just gives you a (more) wrong number. I know which kind of error would be preferred in a video game for instance.
- mbauman 5y ago
  > the other just gives you a (more) wrong number. This is a common misconception: IEEE754 floating point arithmetic is not wrong or fuzzy or inherently inexact. It is very exact; it just has limited precision.
  - chmod775 5y ago
    No. In mathematical terms typical floating point arithmetic has perfect precision - it is repeatable - but limited accuracy - it can deviate from the real value. It's a shortcoming of CS vocabulary that "precision" is also used to refer to the number of bits used. "Exactness" is a fuzzy term used in common english to refer to the concept of both precision and accuracy, or either really. It's not really used in mathematics.
    - mbauman 5y ago
      Oh cool, you want to use different definitions. No matter. Point remains: the floating point representation of 1/4 is not approximate. Similarly, the closest floating point value to 1/10 also precisely defines some value: it's exactly 0.1000000000000000055511151231257827021181583404541015625. Now whether that's accurate to you depends upon your application. But the numbers don't care.

da_chicken 5y ago

> However a point can be made that if you even need to consider such behavior, you should probably be using something more precise than your current number representation. I would agree with this. If you're in a place where you actually need to care about how accurately you're representing numbers with IEEE 754, you almost certainly should look at a precise numeric data type instead and should set aside IEEE 754 entirely. It's less common that we think that we need both the range of values that a float or double represent and need those values represented precisely enough to endure aggregation. It would be nice to be able to give fewer digits to the exponent in a double, though. How often have I really needed 1e308 or 1e-324? How often have I even needed 1e38 or 1e-45? Then again, I don't remember how much precision we'd get back with more digits in the mantissa. Precision is a really funny thing, too. Remember, pi to ~37 decimal places is precise enough to calculate the circumference of a circle of atoms (~10^-10 m) the size of the observable universe (~10^26 m), as at that point the primary source of imprecision is the fact that you used atoms to construct your circle.

tomsmeding 5y ago
> It would be nice to be able to give fewer digits to the exponent in a double, though. How often have I really needed 1e308 or 1e-324? How often have I even needed 1e38 or 1e-45? Then again, I don't remember how much precision we'd get back with more digits in the mantissa. I'm not sure that helps. Going from 10^308 to 10^38, approximately, takes 3 bits from the 11-bit exponent and adds 3 bits to the 52-bit mantissa. This gives you _almost_ one extra decimal digit in your precision. It's not worth it, and if it is you certainly need a more special-purpose number type.