4 ms·

Pretty solid article. I'm glad there was mention of the non-uniform-distribution of floats, though my biased view wishes there were a little more emphasis that

Pretty solid article. I'm glad there was mention of the non-uniform-distribution of floats, though my biased view wishes there were a little more emphasis that floats really really want to be "near 1", to keep precision. The practice of normalizing data to a [0,1] space, doing your math there, then transforming back to "big" space is useful not just for conceptual simplicity, but also for maintaining precision in intermediate results. Also, there's a pretty good range of integers that can be represented exactly by floats; one of the reasons the Lua developers were resistant to adding an integer type for many years. I believe you can represent about 24 bits of exact integers in a 32 bit float. I've been wrangling with fixed-point representations a lot lately, so the differences are stark in my mind (:

deleted 3y ago
[deleted]
Veserv 3y ago
Floating point is a piecewise linear approximation of a exponential function where each piece has fixed precision. For FP32, which has 23 mantissa bits, you divide each piece into 2^23 evenly spaced segments e.g. between 1 and 2 there are 2^23 segments and between 2 and 4 there are 2^23 segments, etc. The strength of floating point is that it bounds % error rather than absolute error. Similar to how scientific notation has a % error represented in the form of number of significant figures.
- mananaysiempre 3y ago
  > The strength of floating point is that it bounds % error rather than absolute error. Similar to how scientific notation has a % error represented in the form of number of significant figures. Not just similar, precisely the same: scientific notation with a fixed number of significant figures is pencil-and-paper decimal floats. Having mentioned decimal floats, though, I have to note that they have worse precision bounds because the pieces in the piecewise approximation are larger. For example, with four binary digits, one unit of last place expressed as a relative error lies from binary 0.001 / 1.111 to binary 0.001 / 1.001, that is to say from about 2^-4 to about 2^-3, a factor of two, whereas with four decimal digits it will be from 0.001 / 9.999 to 0.001 / 1.001, or about 10^-4 to about 10^-3, a factor of ten. The usual phrasing is that the wobble of the number system is smaller with a smaller radix. Trading more wobble for less digits is reasonable when calculating by hand, but makes no sense on a computer. Even on paper some books will tell you to refrain from rounding down to ...1 when a value ends in ...11, ...12, or ...13 in order to avoid the worst of the wobble.
kawogi 3y ago
It's 25 bits if you take the sign bit into account.
seventhson 3y ago
> floats really really want to be "near 1", to keep precision. The number of significant digits is identical for (nearly) the entire range of FP values. There's no value to keeping it "near 1" for IEEE 754 floats - the precision is exactly the same regardless whether near 1 or near 1 trillion. This makes them ideal for general computation and modeling physical properties. In contrast, posits, the unum alternative to IEEE 754, are highly sensitive to absolute scale. Posits lose precision as the magnitudes increase. Otoh, for small values, you get much higher precision which is why they getting some attention from the AI world where normalized weights are everywhere.
- turtledragonfly 3y ago
  Hm, maybe my terminology is not quite right; let me try again: With float32, if I want to do some "small-scale" math with a clump of numbers near 1 trillion, I will have a bad time. I can't even add 1 to them, reliably. Might get x+1==x, and soforth. But If I instead transform my space down to something near 1 (say, subtracting 1 trillion from everything), then I have available the fine-grained results of that math. It's true that when it gets transformed back to "origin = 1trillion", that detail will be lost, but during the time I'm doing those intermediate calculations, the error is staying small in absolute value. So, probably "precision" was the wrong word to use. Maybe relative vs. absolute error? Compare that to fixed-point, where it doesn't matter where the cluster of numbers is, as long as things stay representable (which can indeed be a problem). You'll get the same absolute error either way. Hopefully that makes sense; my grasp of the specific terms, etc is a little tenuous at times (:
  - foota 3y ago
    I feel like I'm having trouble picturing this. Istm like in most cases you're either not taking the big number as an input (in which case you can just make the modification to the big number after the calculation is done) or you are taking the big number as an input, and there's no difference in error. E.g., say you're adding some velocity to an objects position, there no reason to do that "scaled down" since you get garbage in and garbage out regardless? Imo the right advice is to either store things in some sort of offset format e.g., relative to this very far away thing, the position is X meters, or to store the inputs separately e.g., if you're doing position of an object under some constant velocity, store time and velocity separately instead of accumulating it so errors don't add up. But maybe I'm missing something? I guess you'd also want to avoid using larger number in calculations, so e.g., the midpoint trick of adding half the distance between two points to the smaller instead of adding them together and dividing by 2.

deleted 3y ago

[deleted]

Veserv 3y ago

Floating point is a piecewise linear approximation of a exponential function where each piece has fixed precision. For FP32, which has 23 mantissa bits, you divide each piece into 2^23 evenly spaced segments e.g. between 1 and 2 there are 2^23 segments and between 2 and 4 there are 2^23 segments, etc. The strength of floating point is that it bounds % error rather than absolute error. Similar to how scientific notation has a % error represented in the form of number of significant figures.

mananaysiempre 3y ago
> The strength of floating point is that it bounds % error rather than absolute error. Similar to how scientific notation has a % error represented in the form of number of significant figures. Not just similar, precisely the same: scientific notation with a fixed number of significant figures is pencil-and-paper decimal floats. Having mentioned decimal floats, though, I have to note that they have worse precision bounds because the pieces in the piecewise approximation are larger. For example, with four binary digits, one unit of last place expressed as a relative error lies from binary 0.001 / 1.111 to binary 0.001 / 1.001, that is to say from about 2^-4 to about 2^-3, a factor of two, whereas with four decimal digits it will be from 0.001 / 9.999 to 0.001 / 1.001, or about 10^-4 to about 10^-3, a factor of ten. The usual phrasing is that the wobble of the number system is smaller with a smaller radix. Trading more wobble for less digits is reasonable when calculating by hand, but makes no sense on a computer. Even on paper some books will tell you to refrain from rounding down to ...1 when a value ends in ...11, ...12, or ...13 in order to avoid the worst of the wobble.

kawogi 3y ago

It's 25 bits if you take the sign bit into account.

seventhson 3y ago

> floats really really want to be "near 1", to keep precision. The number of significant digits is identical for (nearly) the entire range of FP values. There's no value to keeping it "near 1" for IEEE 754 floats - the precision is exactly the same regardless whether near 1 or near 1 trillion. This makes them ideal for general computation and modeling physical properties. In contrast, posits, the unum alternative to IEEE 754, are highly sensitive to absolute scale. Posits lose precision as the magnitudes increase. Otoh, for small values, you get much higher precision which is why they getting some attention from the AI world where normalized weights are everywhere.

turtledragonfly 3y ago
Hm, maybe my terminology is not quite right; let me try again: With float32, if I want to do some "small-scale" math with a clump of numbers near 1 trillion, I will have a bad time. I can't even add 1 to them, reliably. Might get x+1==x, and soforth. But If I instead transform my space down to something near 1 (say, subtracting 1 trillion from everything), then I have available the fine-grained results of that math. It's true that when it gets transformed back to "origin = 1trillion", that detail will be lost, but during the time I'm doing those intermediate calculations, the error is staying small in absolute value. So, probably "precision" was the wrong word to use. Maybe relative vs. absolute error? Compare that to fixed-point, where it doesn't matter where the cluster of numbers is, as long as things stay representable (which can indeed be a problem). You'll get the same absolute error either way. Hopefully that makes sense; my grasp of the specific terms, etc is a little tenuous at times (:
- foota 3y ago
  I feel like I'm having trouble picturing this. Istm like in most cases you're either not taking the big number as an input (in which case you can just make the modification to the big number after the calculation is done) or you are taking the big number as an input, and there's no difference in error. E.g., say you're adding some velocity to an objects position, there no reason to do that "scaled down" since you get garbage in and garbage out regardless? Imo the right advice is to either store things in some sort of offset format e.g., relative to this very far away thing, the position is X meters, or to store the inputs separately e.g., if you're doing position of an object under some constant velocity, store time and velocity separately instead of accumulating it so errors don't add up. But maybe I'm missing something? I guess you'd also want to avoid using larger number in calculations, so e.g., the midpoint trick of adding half the distance between two points to the smaller instead of adding them together and dividing by 2.