6 ms·

> floats really really want to be "near 1", to keep precision. The number of significant digits is identical for (nearly) the entire range of FP values. There'

> floats really really want to be "near 1", to keep precision. The number of significant digits is identical for (nearly) the entire range of FP values. There's no value to keeping it "near 1" for IEEE 754 floats - the precision is exactly the same regardless whether near 1 or near 1 trillion. This makes them ideal for general computation and modeling physical properties. In contrast, posits, the unum alternative to IEEE 754, are highly sensitive to absolute scale. Posits lose precision as the magnitudes increase. Otoh, for small values, you get much higher precision which is why they getting some attention from the AI world where normalized weights are everywhere.

turtledragonfly 3y ago
Hm, maybe my terminology is not quite right; let me try again: With float32, if I want to do some "small-scale" math with a clump of numbers near 1 trillion, I will have a bad time. I can't even add 1 to them, reliably. Might get x+1==x, and soforth. But If I instead transform my space down to something near 1 (say, subtracting 1 trillion from everything), then I have available the fine-grained results of that math. It's true that when it gets transformed back to "origin = 1trillion", that detail will be lost, but during the time I'm doing those intermediate calculations, the error is staying small in absolute value. So, probably "precision" was the wrong word to use. Maybe relative vs. absolute error? Compare that to fixed-point, where it doesn't matter where the cluster of numbers is, as long as things stay representable (which can indeed be a problem). You'll get the same absolute error either way. Hopefully that makes sense; my grasp of the specific terms, etc is a little tenuous at times (:
- foota 3y ago
  I feel like I'm having trouble picturing this. Istm like in most cases you're either not taking the big number as an input (in which case you can just make the modification to the big number after the calculation is done) or you are taking the big number as an input, and there's no difference in error. E.g., say you're adding some velocity to an objects position, there no reason to do that "scaled down" since you get garbage in and garbage out regardless? Imo the right advice is to either store things in some sort of offset format e.g., relative to this very far away thing, the position is X meters, or to store the inputs separately e.g., if you're doing position of an object under some constant velocity, store time and velocity separately instead of accumulating it so errors don't add up. But maybe I'm missing something? I guess you'd also want to avoid using larger number in calculations, so e.g., the midpoint trick of adding half the distance between two points to the smaller instead of adding them together and dividing by 2.
  - turtledragonfly 3y ago
    > Imo the right advice is to either store things in some sort of offset format Yeah, I agree that's what it comes down to, basically. It's more or less what I meant by "keep things near 1". I almost said "keep things near 0", but I didn't want to suggest that 1e-12 is somehow better than 1e+12. Really, it's more like "try to keep '1 float unit' to be about the size of the stuff you care about in your calculation". > E.g., say you're adding some velocity to an objects position As I interpret this example, I think it would be beneficial to re-center things to a common nearby origin, rather than doing them in some far-away-from-0 space. The more calculations being done (each one introducing some error), the more it matters. For the specific case of "add X to each number" maybe it wouldn't matter, but I'm talking about arbitrary sequences of math being done. If each of your calculations (applying velocity, maybe some rotation, etc) introduces some floating point roundoff error, then that is a potentially large accumulated absolute value when far away from the origin (eg: could be off-by-kilometers rather than off-by-meters). But if you normalize things down to small relative numbers first, produce all your error in that space, then transform back to faraway-space, that's only one opportunity to produce big-size error, rather than several accumulated. In other words, you've taken your local off-by-meters value, transformed it into global kilometer-size space, and so maybe your final result is off-by-1-kilometer, due to some rounding or whatnot. Whereas if you did all your computation in kilometer-space, then each step of math you do can introduce additional off-by-kilometer error, and you could be off-by-10s-of-kilometers in the end. This whole conversation has got me doubting myself, so I made an example in godbolt that hopefully is clear: https://godbolt.org/z/MfP7e7Tbh https://godbolt.org/z/MfP7e7Tbh
- Dylan16807 3y ago
  But that doesn't want to be near 1, it wants you to use the smallest numbers possible. If you're calculating millimeters, you want to be using .001 and .002, not 1.001 and 1.002. I'd phrase it more like: "you lose accuracy when you have multiple numbers that share a big offset compared to their relative scale". Also, to properly consider a trillion in fixed point for a moment: Let's say you have a 44.20 fixed point format, with a range of ±8.8 trillion and a precision of about 1 millionth. Double precision floats will match it in precision around 10 billion. Around the max, floats will have 10 fewer bits of precision. Around 10 million floats will have 10 more bits of precision, around 10 thousand floats will have 20 more bits of precision, etc.
  - turtledragonfly 3y ago
    > If you're calculating millimeters, you want to be using .001 and .002, not 1.001 and 1.002. I think if millimeters are what's important, one should represent them as '1' and '2', no? That's what I meant by keeping things near 1 (apologies for my clumsy language). I mean whatever unit you care about should be approximately "1 unit" in its float representation. But yes, thank you for helping enunciate these things (:
    - ynik 3y ago
      `0.001` and `0.002` are essentially equally accurate as `1` and `2`. `1.001` and `1.002` are worse. In general, multiplicative scaling is useless with floating-point (*); but shifting the coordinate offset (additive, i.e. translation) can be highly useful. You want to move the range of numbers you are dealing with so that is becomes centered around zero (not near 1!). E.g. in Kerbal Space Program, the physics simulation of individual parts within the rocket needs to use a coordinate system centered on the rocket itself; it would be way too inaccurate to use the global coordinate system centered on the sun. (*) The exception is if you need to keep decimal fractions exact, e.g. if dealing with money. In this case, (if a better suited decimal floating-point is unavailable) you want to scale multiplicatively to ensure a cent is 1, not 0.01.
      - turtledragonfly 3y ago
deleted 3y ago
[deleted]

turtledragonfly 3y ago

Hm, maybe my terminology is not quite right; let me try again: With float32, if I want to do some "small-scale" math with a clump of numbers near 1 trillion, I will have a bad time. I can't even add 1 to them, reliably. Might get x+1==x, and soforth. But If I instead transform my space down to something near 1 (say, subtracting 1 trillion from everything), then I have available the fine-grained results of that math. It's true that when it gets transformed back to "origin = 1trillion", that detail will be lost, but during the time I'm doing those intermediate calculations, the error is staying small in absolute value. So, probably "precision" was the wrong word to use. Maybe relative vs. absolute error? Compare that to fixed-point, where it doesn't matter where the cluster of numbers is, as long as things stay representable (which can indeed be a problem). You'll get the same absolute error either way. Hopefully that makes sense; my grasp of the specific terms, etc is a little tenuous at times (:

foota 3y ago
I feel like I'm having trouble picturing this. Istm like in most cases you're either not taking the big number as an input (in which case you can just make the modification to the big number after the calculation is done) or you are taking the big number as an input, and there's no difference in error. E.g., say you're adding some velocity to an objects position, there no reason to do that "scaled down" since you get garbage in and garbage out regardless? Imo the right advice is to either store things in some sort of offset format e.g., relative to this very far away thing, the position is X meters, or to store the inputs separately e.g., if you're doing position of an object under some constant velocity, store time and velocity separately instead of accumulating it so errors don't add up. But maybe I'm missing something? I guess you'd also want to avoid using larger number in calculations, so e.g., the midpoint trick of adding half the distance between two points to the smaller instead of adding them together and dividing by 2.
- turtledragonfly 3y ago
  > Imo the right advice is to either store things in some sort of offset format Yeah, I agree that's what it comes down to, basically. It's more or less what I meant by "keep things near 1". I almost said "keep things near 0", but I didn't want to suggest that 1e-12 is somehow better than 1e+12. Really, it's more like "try to keep '1 float unit' to be about the size of the stuff you care about in your calculation". > E.g., say you're adding some velocity to an objects position As I interpret this example, I think it would be beneficial to re-center things to a common nearby origin, rather than doing them in some far-away-from-0 space. The more calculations being done (each one introducing some error), the more it matters. For the specific case of "add X to each number" maybe it wouldn't matter, but I'm talking about arbitrary sequences of math being done. If each of your calculations (applying velocity, maybe some rotation, etc) introduces some floating point roundoff error, then that is a potentially large accumulated absolute value when far away from the origin (eg: could be off-by-kilometers rather than off-by-meters). But if you normalize things down to small relative numbers first, produce all your error in that space, then transform back to faraway-space, that's only one opportunity to produce big-size error, rather than several accumulated. In other words, you've taken your local off-by-meters value, transformed it into global kilometer-size space, and so maybe your final result is off-by-1-kilometer, due to some rounding or whatnot. Whereas if you did all your computation in kilometer-space, then each step of math you do can introduce additional off-by-kilometer error, and you could be off-by-10s-of-kilometers in the end. This whole conversation has got me doubting myself, so I made an example in godbolt that hopefully is clear: https://godbolt.org/z/MfP7e7Tbh https://godbolt.org/z/MfP7e7Tbh
Dylan16807 3y ago
But that doesn't want to be near 1, it wants you to use the smallest numbers possible. If you're calculating millimeters, you want to be using .001 and .002, not 1.001 and 1.002. I'd phrase it more like: "you lose accuracy when you have multiple numbers that share a big offset compared to their relative scale". Also, to properly consider a trillion in fixed point for a moment: Let's say you have a 44.20 fixed point format, with a range of ±8.8 trillion and a precision of about 1 millionth. Double precision floats will match it in precision around 10 billion. Around the max, floats will have 10 fewer bits of precision. Around 10 million floats will have 10 more bits of precision, around 10 thousand floats will have 20 more bits of precision, etc.
- turtledragonfly 3y ago
  > If you're calculating millimeters, you want to be using .001 and .002, not 1.001 and 1.002. I think if millimeters are what's important, one should represent them as '1' and '2', no? That's what I meant by keeping things near 1 (apologies for my clumsy language). I mean whatever unit you care about should be approximately "1 unit" in its float representation. But yes, thank you for helping enunciate these things (:
  - ynik 3y ago
    `0.001` and `0.002` are essentially equally accurate as `1` and `2`. `1.001` and `1.002` are worse. In general, multiplicative scaling is useless with floating-point (*); but shifting the coordinate offset (additive, i.e. translation) can be highly useful. You want to move the range of numbers you are dealing with so that is becomes centered around zero (not near 1!). E.g. in Kerbal Space Program, the physics simulation of individual parts within the rocket needs to use a coordinate system centered on the rocket itself; it would be way too inaccurate to use the global coordinate system centered on the sun. (*) The exception is if you need to keep decimal fractions exact, e.g. if dealing with money. In this case, (if a better suited decimal floating-point is unavailable) you want to scale multiplicatively to ensure a cent is 1, not 0.01.
    - turtledragonfly 3y ago

deleted 3y ago

[deleted]