3 ms·
Here's everything you need to know about Floating Point in as shortly as I can write it. 1. Floating points are simply "Binary Scientific notation". The speed
by dragontamer 9y ago
Here's everything you need to know about Floating Point in as shortly as I can write it.
1. Floating points are simply "Binary Scientific notation". The speed of light is 2.98E8... which in "normal form" is written 298,000,000. An IEEE 754 Single has 8-bits for the exponent (E8 in the speed of light), and 24-bits for the mantissa (the 2.98 part). There's some complicated stuff like offset shifting here, but this is the "core idea" of floating point.
2. "Rounding" is forced to happen in Floating Point whenever "information drops off" the far side of the mantissa. The mantissa is only 24-bits long, and many numbers (such as .1) require an infinite number of bits to represent! As such, this "rounding error" builds up exponentially the more operations you perform.
3. Subtraction (cancellation error) is the biggest single source of error and the one that needs to be most studied. "Subtraction" can occur when a positive and negative number is added together.
4. Because of this error (and all errors!), Floating point operations are NOT associative. (A + B) + C gives a different value than A + (B + C). The commutative property remains for multiplication and addition (A+B == B+A). If you require "bit-perfect" and consistent floating-point simulations, you MUST take into account the order of all operations, even simple addition and multiplication.
For example: Try "0.1 + 0.7 + 1" vs "1 + 0.1 + .7" in Python, and you'll see that these to orderings lead to different results.
---------------
Once you fully know and understand these 4 facts, then everything else is just icing on the cake. For example to prevent "cancellation error" (#3), you can sort the numbers by magnitude, and then add them up from smallest magnitude to largest magnitude.
- seattleeng 9y agoThis is an excellent short list of the main points I learned as an undergrad, and what I've retained today. One other point is that 64 bit floating points (aka doubles) really are about double the precision of 32 bit floating points, in terms of significant digits. An interesting implementation detail is that they spend a higher fraction of their "bit budget" on the mantissa to get this (52 out of 64 bits, vs 23 out of 32).
- deleted 9y ago[deleted]
- angersock 9y ago23 bits for the mantissa, and you're forgetting the leading sign bit.
- conistonwater 9y agoNo, it's a 24 bits significand, you're forgetting the not-explicitly-stored leading 1. (https://en.wikipedia.org/wiki/Single-precision_floating-point_format https://en.wikipedia.org/wiki/Single-precision_floating-poin...)
- dnautics 9y agoAre you sure rounding error builds up exponentially? I may be wrong but I thought it was sqrt(N) where N is the number of operations.
- dragontamer 9y agoIt has been a long time since I've taken the class. But my understanding is that the absolute error builds up exponentially. We both might be still correct however: sqrt(N) sounds about the speed of the "number of bits of error". So it depends on how you're counting error. If you're counting "bits of error", then sqrt(N) sounds right. If you're counting absolute error, then I'm pretty sure its exponential.