6 ms·

Myths About Floating-Point Numbers (2021)

66 points by Bogdanp 1y ago

BugsJustFindMe 1y ago
I've never heard anyone say any of these supposed myths, except for the first one, sort of, but nobody means what the first one pretends it means, so this whole post feels like a big strawman to me.
- rollcat 1y ago
  People do say a lot of nonsense about floats, my younger self trying to bash JavaScript included. They e.g. do fine as integers - up to 2**53; this could be optimised by a JIT to use actual integer math.
- Cieric 1y ago
  Just to add some context Adam works at AMD as a dev tech, so he is constantly working with game studios developers directly. While I can't say I've heard the same things since I'm somewhere else, I have seen some of the assumptions made in some shader code and they do line up with the kind of things he's saying.
- jbjbjbjb 1y ago
  I was putting together some technical interview questions for our candidates and I wanted to see what ChatGPT would put for an answer. It told me floating point numbers were non-deterministic and wouldn’t back down from that answer so it’s getting it from somewhere.
- AshamedCaptain 1y ago
  I concur, it seems low effort, and the only real common "myth" (the 1st one) is not really disproven. Infact the very example he puts goes to prove it, as it is going to become an infinite loop given large enough N.... Also compiler optimizations should not affect the result. Specially without "fast-math/O3" (which arguably many people stupidly use nowadays, then complain).
  - zokier 1y ago
    > Also compiler optimizations should not affect the result. Specially without "fast-math/O3" (which arguably many people stupidly use nowadays, then complain). -ffp-contract is annoying exception to that principle.
- jandrese 1y ago
  The first rule is true, but relying on it is dangerous unless you are well versed in what floats can and can not be represented exactly. It's best to pretend it isn't true in most cases to avoid footguns.
- janalsncm 1y ago
  Same, but I still learned a fair amount anyways. I say no harm, no foul.
- nspattak 1y ago
  i am laughing and crying at the same time... a couple of decades ago, I was still a lowly applied mathematics (and aspiring programmer) student when i proposed to a professor to change the compiler options from O2 to O3 to gain performance (reducing run time by a significant percentage like 50% or more...) only to be disregarded by saying "no, compiler optimizations break numerical accuracy".
rollcat 1y ago
Related: What Every Computer Scientist Should Know About Floating-Point Arithmetic <https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.h...>
- hans_castorp 1y ago
  Also related: https://floating-point-gui.de/ https://floating-point-gui.de/
  - zahlman 1y ago
    My list includes: https://0.30000000000000004.com/ https://0.30000000000000004.com/
bee_rider 1y ago
I like these. They are push back against the sort of… first correction that people make when encountering floating point weirdness. That is, the first mistake we make is to treat floats as reals, and then we observe some odd rounding behavior. The second mistake we make is to treat the rounding events as random. A nice thing about IEEE floats is that the rounding behavior is well defined. Often it doesn't matter, like you ask for a gemm and you get whatever order of operations blas, AVX-whatever, and OpenMP conspire to give you, so it is more-or-less random. But if it does matter, the ability to define it is there.
- jandrese 1y ago
  > A nice thing about IEEE floats is that the rounding behavior is well defined. Until it isn't. I used to play the CodeWeavers port of Kohan and while the game would allow you to do crossplay between Windows and Linux, differences in how the two OSes rounded floats would cause the game to desynchronize after 15-20 minutes of play or so. Some unit's pathfinding algorithm would zig on Windows and zag on Linux, causing the state to diverge and end up kicking off one of the players.
  - bee_rider 1y ago
    Is it possible that your different operating systems just had different mxcsr values? Or, since it was a port, maybe they were compiled with different optimizations. There are a lot of things happening under the hood but most of them should be deterministic.
    - toolslive 1y ago
      until someone compiles with --ffast-math enabled, stating "I don't care about accuracy, as long as it's fast".
      - bee_rider 1y ago
        It is good to enable that flag because it also enables the “fun safe math optimizations” flag, and it is important to remind people that math is a safe way to have fun.
exDM69 1y ago
My favorite thing about floating point numbers: you can divide by zero. The result of x/0.0 is +/- inf (or NaN if x is zero). There's a helpful table in "weird floats" [0] that covers all the cases for division and a bunch of other arithmetic instructions. This is especially useful when writing branchless or SIMD code. Adding a branch for checking against zero can have bad performance implications and it isn't even necessary in many cases. Especially in graphics code I often see a zero check before a division and a fallback for the zero case. This often practically means "wait until numerical precision artifacts arise and then do something else". Often you could just choose the better of the two options you have instead of checking for zero. Case in point: choosing the axes of your shadow map projection matrix. You have two options (world x axis or z axis), choose the better one (larger angle with viewing direction). Don't wait until the division goes to inf and then fall back to the other. [0] https://www.cs.uaf.edu/2011/fall/cs301/lecture/11_09_weird_floats.html https://www.cs.uaf.edu/2011/fall/cs301/lecture/11_09_weird_f...
- thwarted 1y ago
  You still can't divide by zero, it just doesn't result in an error state that stops execution. The inf and NaN values are sentinel values that you still have to check for after the calculation to know if it went awry.
  - sixo 1y ago
    In the space of floats, you are dividing by zero. To map back to the space of numbers you have to check. It's nice, though; inf and NaN sentinels give you the behavior of a monadic `Result | Error` pipeline without having to wrap your numbers in another abstraction.
  - TehShrike 1y ago
    You can divide by zero, but you mayn't.
  - epcoa 1y ago
    If dividing by zero has a well defined result that doesn’t abort execution what exactly does “can’t” even mean? Operations on those sentinel values are also defined. This can affect when checking needs to be done in optimized code.
BlackFly 1y ago
You can exploit the exactness of (specific) floating point operations in test data by using sums of powers of 2. Polynomials with such coefficients produce exact results so long as the overall powers are within ~53 powers of 2 (don't quote me exactly on that, I generally don't push the range very high!). You can find exact polynomial solutions to linear PDEs with such powers using high enough order finite difference methods for example. However, the story about non-determinism is no myth. The intel processors have a separate math coprocessor that supports 80bit floats (https://en.wikipedia.org/wiki/Extended_precision#x86_extended-precision_format https://en.wikipedia.org/wiki/Extended_precision#x86_extende...). Moving a float from a register in this coprocessor to memory truncates the float. Repeated math can be done inside this coprocessor to achieve higher precision so hot loops generally don't move floats outside of these registers. Non-determinism occurs in programs running on intel with floats when threads are interrupted and the math coprocessor flushed. The non-determinism isn't intrinsic to the floating point arithmetic but to the non-determinism of when this truncation may occur. This is more relevant for fields where chaotic dynamics occur. So the same program with the same inputs can produce different results. NaN is an error. If you take the square root of a negative number you get a NaN. This is just a type error, use complex numbers to overcome this one. But then you get 0. / 0. and that's a NaN or Inf - Inf and a whole slew of other things that produce out of bounds results. Whether it is expected or not is another story, but it does mean that you are unable to represent the value with a float and that is a type error.
- AshamedCaptain 1y ago
  > Non-determinism occurs in programs running on intel with floats when threads are interrupted and the math coprocessor flushed That's ridiculous. No OS in his right mind would flush FPU regs to 64 bits only, because that would break many things, most obviously "real" 80 bit FP which is still a thing and the only reason x87 instructions still work. It would even break plain equality comparisons making all FP useless. For 64 bit FP most compilers prefer SSE rather than x87 instructions these days.
  - dur-randir 1y ago
    https://bugs.php.net/bug.php?id=53632 https://bugs.php.net/bug.php?id=53632 Never, for sure.
quietbritishjim 1y ago
When writing code, it's useful to follow simpler rules than strictly needed to make it easier to read for other coders (including you in the future). For example, you might add some unnecessary brackets to expressions to avoid remembering all the operator precedence rules. The article is right that sometimes you can can use simple equality on floating point values. But if you have a (almost) blanket rule to use fuzzy comparison, or (even better) just avoid any sort if equality-like comparison altogether, then your code might be simpler to understand and you might be less likely to make a mistake about when you really can safely do that. It's still sensible to understand that it's possible though. Just that you should try to avoid writing tricky code that depends on it.
- zokier 1y ago
  That sort of argument can be problematic though, because the code can become misleading. If I see a fuzzy comparison then I'd assume it is there because it is needed, which might make the rest of the code more difficult to understand/modify because I then have to assume that everywhere else the values might be fuzzy.
- kccqzy 1y ago
  Every time I use exact floating point equality I simply write a comment explaining why I did this. That's what comments are for.
glkindlmann 1y ago
Note: the improved loop in the "1. They are not exact" can easily hang. If count > 2^24, then the ulp of f is 2, and adding 1.0f leaves f unchanged. What's wild is that a few lines later he notes how above 2^24 numbers start “jumping” every 2. Ok, but FP are always "jumping", by their ulp, regardless of how big or small they are.
pklausler 1y ago
People need to understand rounding better, especially the topic of when rounding can happen and when it can't for the basic operations. Updating with a concrete example: The Fortran standard defines the real MOD and MODULO intrinsic functions as being equivalent to the straightforward sequence of a division, conversion to integer, multiplication, and subtraction. This formula can round, obviously. But MOD can be (and thus should be) implemented exactly by other means, and most Fortran compilers do so instead. This leaves Fortran implementors in a bit of a pickle -- conform to the standard, or produce good results?
- camgunz 1y ago
  Definitely. People think they'll get out of knowing how rounding works by using arbitrary precision arithmetic, but arguably it's even more important there (you run out of precision/memory at some point; what do you think happens then?). You can use floats for money if you do the rounding right.
AtlasBarfed 1y ago
The core thing to know about floating point numbers in comp langs is they aren't floating point numbers. They are approximations of floating point numbers, and even if your current approximation of a floating point number to represent a value seems to be accurate as an integer vale... There is no guarantee if you take two of those floating point number approximations that appear completely accurate that the resulting operation between them will also be completely accurate.
- kccqzy 1y ago
  That's not a useful way to think. Floating point numbers are just floating point numbers. You aren't approximating floating point numbers. You are approximating real numbers. Floating point numbers are compared against fixed point numbers where the point (the part between the integer part and fractional part) is fixed. That is why they are called floating. The nature of the real numbers is such that they can in general only be approximated, regardless of whether you use fixed point numbers or floating point numbers or a fancier computable number representation.
  - ogogmad 1y ago
    The real numbers can be "represented" and "computed with" without needing approximations. The result is that there are no rounding errors. But it's not popular, and not necessarily worth the tradeoffs. On the other hand, see the Android calculator.
    - kccqzy 1y ago
      The Android calculator doesn't support all real numbers. It supports a specific subset of computable numbers that are computable using the operations presented in the calculator app. It's a good domain-specific number representation. It's far from real numbers. We can't even represent all the natural numbers: that would require infinite memory. Now you want to represent the real numbers, equinumerous with the power set of natural numbers?
      - 1y ago
ivankra 1y ago
My favorite trick: NaN boxing. NaN's aren't just for errors, but also for smuggling other data inside. For a double, you have whopping 53 bits of payload, enough to cram in a pointer and maybe a type tag, and many javascript engines do (since JS numbers are double's after all) https://wingolog.org/archives/2011/05/18/value-representation-in-javascript-implementations https://wingolog.org/archives/2011/05/18/value-representatio... https://piotrduperas.com/posts/nan-boxing https://piotrduperas.com/posts/nan-boxing
- pklausler 1y ago
  52 bits of payload, and at least one bit must be set.
  - ivankra 1y ago
    You can put stuff into the sign bit too, that makes 53. Yeah, the lower 52 bits can't all be zero - that'd be ±INF, but the other 2^53-2 values are all yours to use.
    - pklausler 1y ago
      It's possible for the sign bit of a NaN to be changed by a "non-arithmetic" operation that doesn't trap on the NaN, so don't put anything precious in there.
- Arnavion 1y ago
  It is also how RISC-V floating point registers are required to store floats of smaller widths. Eg if your CPU supports 64-bit floats (D extension), its FPU registers will be 64-bit wide. If you use an instruction to load a 16-bit float (Zfh extension) into such a register, it will be boxed into a negative quiet NaN with all bits above the lower 16 bits set to 1.
calibas 1y ago
> They are not exact It's not exactly a myth, as the article mentions, they're only exact for certain ranges of values. > NaN and INF are indication of an error This is somewhat semantic, but dividing by zero typically does create a hardware exception. However, it's all handled behind the scenes, and you get "Inf" as the result. You can make it so dividing by zero is explicitly an error, see the "ftrapping-math" flag.
- deleted 1y ago
  [deleted]
- GuB-42 1y ago
  I think the myth about exactness is that you can't use strict equality with floating point numbers because they are somehow fuzzy. They are not. Some operations involve rounding, notably conversion from decimal, but the (rounded) result is an exact number that can be stored and recovered without loss of precision and equality will work as expected. Converting from floating point binary to decimal is exact though, given enough digits (can be >1000 for very small or very large numbers).
  - Joker_vD 1y ago
    > you can't use strict equality with floating point numbers because they are somehow fuzzy. They are not. They are though. All arithmetic operations involve rounding, so e.g. (7.0 / 1234 + 0.5) * 1234 is not equal to 7.0 + 617 (it differs in 1 ULP). On the other hand, (9.0 / 1234 + 0.5) * 1234 is equal to 9.0 + 617, so the end result is sometimes exact and sometimes is not. How can you know beforehand which one is the case in your specific case? Generally, you can't, any arithmetic operation can potentially give you 1 ULP of error, and it can (and likely, will) slowly accumulate.
AndriyKunitsyn 1y ago
Here's one that's not a myth: IEEE-754 floats are the only "primitive types" that allow "a == a" to not be true. I.e., two floats that are _identical_ to each other (even when it's _the same_ variable, on the same memory address) can be not _equal_ to each other, specifically if it's NaN. This is dictated by IEEE-754, and this is true for all programming languages I know, and to this day, this makes zero sense to me, but apparently this is useful for some reason.
- zahlman 1y ago
  >but apparently this is useful for some reason. It lets you easily test whether a value is NaN without needing a library function call. (Even if the language wanted to provide a NaN literal, there's more than one NaN value, so that wouldn't actually work!)
  - AndriyKunitsyn 1y ago
    This test is better with a function call, because it clearly shows the intent. Perf-wise, this function call will be inlined by any decent compiler. https://godbolt.org/z/chhM85vP3 https://godbolt.org/z/chhM85vP3 Breaking a basic programming expectation for a specific class of values of a specific type is not a good thing, in my opinion.
analog31 1y ago
>>> A beginner would use them, trusting they are infinitely capable and precise, which can lead to problems. An intermediate programmer knows that they have some limitations, and so by using some good practices the problems can be avoided. Amusingly, pocket calculators became affordable while I was in high school, and anybody who was interested in math learned the foibles of floating point almost instantly. Now it's an advanced topic. A difference may have been that our calculators had very few significant digits, so the errors introduced by successive calculations became noticeable more quickly. Also, you had to think about what you were doing because, even if you trusted the calculator, you didn't trust your fingers.
- pclmulqdq 1y ago
  Modern pocket calculators now use some very fancy math that makes them have much less error than if they used floating point. Many of them will use arbitrary-precision arithmetic at minimum, and there are some fancier schemes.
fiforpg 1y ago
In the context of double precision the article says > the largest integer value that can be represented exactly is 2^53 — I am confused as to why it not 2^52, given that there are 52 bits of mantissa, so relative accuracy is 2^-52, which translates to absolute accuracy larger than 1 after 2^52. Compare this to the table there saying "Next value after 1 = 1 + 2^-52".
- pclmulqdq 1y ago
  There's an implied one bit, so you actually have a 53 bit significand (and 53-bit precision) given only a 52 bit mantissa.
  - fiforpg 1y ago
    Right, I did realize after posting that close to numbers of the form 1{hidden bit} + (1-2^-52){mantissa with all ones} the relative accuracy — corresponding to the absolute accuracy of a single bit in mantissa — is about 2^-53. The hidden bit is easy to forget about...
- layer8 1y ago
  https://stackoverflow.com/questions/1848700/biggest-integer-that-can-be-stored-in-a-double https://stackoverflow.com/questions/1848700/biggest-integer-...
layer8 1y ago
Archive of the defunct tweet link: https://web.archive.org/web/20210608061416/https://twitter.com/j_bikker/status/1372273107197698054 https://web.archive.org/web/20210608061416/https://twitter.c...
zX41ZdbW 1y ago
This is a great article! A few more resources on this topic: https://randomascii.wordpress.com/category/floating-point/ https://randomascii.wordpress.com/category/floating-point/ - a series of articles about floating point, with deeper details. https://float.exposed/ https://float.exposed/ - explorer of floating point values by Bartosz Ciechanowski.
- zX41ZdbW 1y ago
  Fun facts about floating point, from my practice: Obvious: An array containing NaNs will almost certainly be sorted incorrectly with std::sort. Non-obvious: sorting an array containing NaNs with std::sort from libstdc++ (gcc's implementation) could lead to a buffer overflow. Non-obvious: using MMX instructions without clearing the FPU state leads to incorrect results of floating-point instructions in other parts of the code base: https://github.com/simdjson/simdjson/issues/169 https://github.com/simdjson/simdjson/issues/169
- extraduder_ire 1y ago
  I love that float.exposed website. Really nails home a point I saw made a few years ago about how there's only a few billion possible floats, so you can often test them all.
vismit2000 1y ago
A famous old interview question regarding floating points: https://stackoverflow.com/questions/6699066/in-which-order-should-floats-be-added-to-get-the-most-precise-result https://stackoverflow.com/questions/6699066/in-which-order-s... This awesome article drives home the point very succinctly: https://fabiensanglard.net/floating_point_visually_explained/ https://fabiensanglard.net/floating_point_visually_explained...

66 points by Bogdanp 1y ago

I've never heard anyone say any of these supposed myths, except for the first one, sort of, but nobody means what the first one pretends it means, so this whole post feels like a big strawman to me.

rollcat 1y ago
People do say a lot of nonsense about floats, my younger self trying to bash JavaScript included. They e.g. do fine as integers - up to 2**53; this could be optimised by a JIT to use actual integer math.
Cieric 1y ago
Just to add some context Adam works at AMD as a dev tech, so he is constantly working with game studios developers directly. While I can't say I've heard the same things since I'm somewhere else, I have seen some of the assumptions made in some shader code and they do line up with the kind of things he's saying.
jbjbjbjb 1y ago
I was putting together some technical interview questions for our candidates and I wanted to see what ChatGPT would put for an answer. It told me floating point numbers were non-deterministic and wouldn’t back down from that answer so it’s getting it from somewhere.
AshamedCaptain 1y ago
I concur, it seems low effort, and the only real common "myth" (the 1st one) is not really disproven. Infact the very example he puts goes to prove it, as it is going to become an infinite loop given large enough N.... Also compiler optimizations should not affect the result. Specially without "fast-math/O3" (which arguably many people stupidly use nowadays, then complain).
- zokier 1y ago
  > Also compiler optimizations should not affect the result. Specially without "fast-math/O3" (which arguably many people stupidly use nowadays, then complain). -ffp-contract is annoying exception to that principle.
jandrese 1y ago
The first rule is true, but relying on it is dangerous unless you are well versed in what floats can and can not be represented exactly. It's best to pretend it isn't true in most cases to avoid footguns.
janalsncm 1y ago
Same, but I still learned a fair amount anyways. I say no harm, no foul.
nspattak 1y ago
i am laughing and crying at the same time... a couple of decades ago, I was still a lowly applied mathematics (and aspiring programmer) student when i proposed to a professor to change the compiler options from O2 to O3 to gain performance (reducing run time by a significant percentage like 50% or more...) only to be disregarded by saying "no, compiler optimizations break numerical accuracy".

rollcat 1y ago

Related: What Every Computer Scientist Should Know About Floating-Point Arithmetic <https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.h...>

hans_castorp 1y ago
Also related: https://floating-point-gui.de/ https://floating-point-gui.de/
- zahlman 1y ago
  My list includes: https://0.30000000000000004.com/ https://0.30000000000000004.com/

bee_rider 1y ago

I like these. They are push back against the sort of… first correction that people make when encountering floating point weirdness. That is, the first mistake we make is to treat floats as reals, and then we observe some odd rounding behavior. The second mistake we make is to treat the rounding events as random. A nice thing about IEEE floats is that the rounding behavior is well defined. Often it doesn't matter, like you ask for a gemm and you get whatever order of operations blas, AVX-whatever, and OpenMP conspire to give you, so it is more-or-less random. But if it does matter, the ability to define it is there.

jandrese 1y ago
> A nice thing about IEEE floats is that the rounding behavior is well defined. Until it isn't. I used to play the CodeWeavers port of Kohan and while the game would allow you to do crossplay between Windows and Linux, differences in how the two OSes rounded floats would cause the game to desynchronize after 15-20 minutes of play or so. Some unit's pathfinding algorithm would zig on Windows and zag on Linux, causing the state to diverge and end up kicking off one of the players.
- bee_rider 1y ago
  Is it possible that your different operating systems just had different mxcsr values? Or, since it was a port, maybe they were compiled with different optimizations. There are a lot of things happening under the hood but most of them should be deterministic.
  - toolslive 1y ago
    until someone compiles with --ffast-math enabled, stating "I don't care about accuracy, as long as it's fast".
    - bee_rider 1y ago
      It is good to enable that flag because it also enables the “fun safe math optimizations” flag, and it is important to remind people that math is a safe way to have fun.

exDM69 1y ago

My favorite thing about floating point numbers: you can divide by zero. The result of x/0.0 is +/- inf (or NaN if x is zero). There's a helpful table in "weird floats" [0] that covers all the cases for division and a bunch of other arithmetic instructions. This is especially useful when writing branchless or SIMD code. Adding a branch for checking against zero can have bad performance implications and it isn't even necessary in many cases. Especially in graphics code I often see a zero check before a division and a fallback for the zero case. This often practically means "wait until numerical precision artifacts arise and then do something else". Often you could just choose the better of the two options you have instead of checking for zero. Case in point: choosing the axes of your shadow map projection matrix. You have two options (world x axis or z axis), choose the better one (larger angle with viewing direction). Don't wait until the division goes to inf and then fall back to the other. [0] https://www.cs.uaf.edu/2011/fall/cs301/lecture/11_09_weird_floats.html https://www.cs.uaf.edu/2011/fall/cs301/lecture/11_09_weird_f...

thwarted 1y ago
You still can't divide by zero, it just doesn't result in an error state that stops execution. The inf and NaN values are sentinel values that you still have to check for after the calculation to know if it went awry.
- sixo 1y ago
  In the space of floats, you are dividing by zero. To map back to the space of numbers you have to check. It's nice, though; inf and NaN sentinels give you the behavior of a monadic `Result | Error` pipeline without having to wrap your numbers in another abstraction.
- TehShrike 1y ago
  You can divide by zero, but you mayn't.
- epcoa 1y ago
  If dividing by zero has a well defined result that doesn’t abort execution what exactly does “can’t” even mean? Operations on those sentinel values are also defined. This can affect when checking needs to be done in optimized code.

BlackFly 1y ago

You can exploit the exactness of (specific) floating point operations in test data by using sums of powers of 2. Polynomials with such coefficients produce exact results so long as the overall powers are within ~53 powers of 2 (don't quote me exactly on that, I generally don't push the range very high!). You can find exact polynomial solutions to linear PDEs with such powers using high enough order finite difference methods for example. However, the story about non-determinism is no myth. The intel processors have a separate math coprocessor that supports 80bit floats (https://en.wikipedia.org/wiki/Extended_precision#x86_extended-precision_format https://en.wikipedia.org/wiki/Extended_precision#x86_extende...). Moving a float from a register in this coprocessor to memory truncates the float. Repeated math can be done inside this coprocessor to achieve higher precision so hot loops generally don't move floats outside of these registers. Non-determinism occurs in programs running on intel with floats when threads are interrupted and the math coprocessor flushed. The non-determinism isn't intrinsic to the floating point arithmetic but to the non-determinism of when this truncation may occur. This is more relevant for fields where chaotic dynamics occur. So the same program with the same inputs can produce different results. NaN is an error. If you take the square root of a negative number you get a NaN. This is just a type error, use complex numbers to overcome this one. But then you get 0. / 0. and that's a NaN or Inf - Inf and a whole slew of other things that produce out of bounds results. Whether it is expected or not is another story, but it does mean that you are unable to represent the value with a float and that is a type error.

AshamedCaptain 1y ago
> Non-determinism occurs in programs running on intel with floats when threads are interrupted and the math coprocessor flushed That's ridiculous. No OS in his right mind would flush FPU regs to 64 bits only, because that would break many things, most obviously "real" 80 bit FP which is still a thing and the only reason x87 instructions still work. It would even break plain equality comparisons making all FP useless. For 64 bit FP most compilers prefer SSE rather than x87 instructions these days.
- dur-randir 1y ago
  https://bugs.php.net/bug.php?id=53632 https://bugs.php.net/bug.php?id=53632 Never, for sure.

quietbritishjim 1y ago

When writing code, it's useful to follow simpler rules than strictly needed to make it easier to read for other coders (including you in the future). For example, you might add some unnecessary brackets to expressions to avoid remembering all the operator precedence rules. The article is right that sometimes you can can use simple equality on floating point values. But if you have a (almost) blanket rule to use fuzzy comparison, or (even better) just avoid any sort if equality-like comparison altogether, then your code might be simpler to understand and you might be less likely to make a mistake about when you really can safely do that. It's still sensible to understand that it's possible though. Just that you should try to avoid writing tricky code that depends on it.

zokier 1y ago
That sort of argument can be problematic though, because the code can become misleading. If I see a fuzzy comparison then I'd assume it is there because it is needed, which might make the rest of the code more difficult to understand/modify because I then have to assume that everywhere else the values might be fuzzy.
kccqzy 1y ago
Every time I use exact floating point equality I simply write a comment explaining why I did this. That's what comments are for.

glkindlmann 1y ago

Note: the improved loop in the "1. They are not exact" can easily hang. If count > 2^24, then the ulp of f is 2, and adding 1.0f leaves f unchanged. What's wild is that a few lines later he notes how above 2^24 numbers start “jumping” every 2. Ok, but FP are always "jumping", by their ulp, regardless of how big or small they are.

pklausler 1y ago

People need to understand rounding better, especially the topic of when rounding can happen and when it can't for the basic operations. Updating with a concrete example: The Fortran standard defines the real MOD and MODULO intrinsic functions as being equivalent to the straightforward sequence of a division, conversion to integer, multiplication, and subtraction. This formula can round, obviously. But MOD can be (and thus should be) implemented exactly by other means, and most Fortran compilers do so instead. This leaves Fortran implementors in a bit of a pickle -- conform to the standard, or produce good results?

camgunz 1y ago
Definitely. People think they'll get out of knowing how rounding works by using arbitrary precision arithmetic, but arguably it's even more important there (you run out of precision/memory at some point; what do you think happens then?). You can use floats for money if you do the rounding right.

AtlasBarfed 1y ago

The core thing to know about floating point numbers in comp langs is they aren't floating point numbers. They are approximations of floating point numbers, and even if your current approximation of a floating point number to represent a value seems to be accurate as an integer vale... There is no guarantee if you take two of those floating point number approximations that appear completely accurate that the resulting operation between them will also be completely accurate.

kccqzy 1y ago
That's not a useful way to think. Floating point numbers are just floating point numbers. You aren't approximating floating point numbers. You are approximating real numbers. Floating point numbers are compared against fixed point numbers where the point (the part between the integer part and fractional part) is fixed. That is why they are called floating. The nature of the real numbers is such that they can in general only be approximated, regardless of whether you use fixed point numbers or floating point numbers or a fancier computable number representation.
- ogogmad 1y ago
  The real numbers can be "represented" and "computed with" without needing approximations. The result is that there are no rounding errors. But it's not popular, and not necessarily worth the tradeoffs. On the other hand, see the Android calculator.
  - kccqzy 1y ago
    The Android calculator doesn't support all real numbers. It supports a specific subset of computable numbers that are computable using the operations presented in the calculator app. It's a good domain-specific number representation. It's far from real numbers. We can't even represent all the natural numbers: that would require infinite memory. Now you want to represent the real numbers, equinumerous with the power set of natural numbers?
    - 1y ago

ivankra 1y ago

My favorite trick: NaN boxing. NaN's aren't just for errors, but also for smuggling other data inside. For a double, you have whopping 53 bits of payload, enough to cram in a pointer and maybe a type tag, and many javascript engines do (since JS numbers are double's after all) https://wingolog.org/archives/2011/05/18/value-representation-in-javascript-implementations https://wingolog.org/archives/2011/05/18/value-representatio... https://piotrduperas.com/posts/nan-boxing https://piotrduperas.com/posts/nan-boxing

pklausler 1y ago
52 bits of payload, and at least one bit must be set.
- ivankra 1y ago
  You can put stuff into the sign bit too, that makes 53. Yeah, the lower 52 bits can't all be zero - that'd be ±INF, but the other 2^53-2 values are all yours to use.
  - pklausler 1y ago
    It's possible for the sign bit of a NaN to be changed by a "non-arithmetic" operation that doesn't trap on the NaN, so don't put anything precious in there.
Arnavion 1y ago
It is also how RISC-V floating point registers are required to store floats of smaller widths. Eg if your CPU supports 64-bit floats (D extension), its FPU registers will be 64-bit wide. If you use an instruction to load a 16-bit float (Zfh extension) into such a register, it will be boxed into a negative quiet NaN with all bits above the lower 16 bits set to 1.

calibas 1y ago

> They are not exact It's not exactly a myth, as the article mentions, they're only exact for certain ranges of values. > NaN and INF are indication of an error This is somewhat semantic, but dividing by zero typically does create a hardware exception. However, it's all handled behind the scenes, and you get "Inf" as the result. You can make it so dividing by zero is explicitly an error, see the "ftrapping-math" flag.

deleted 1y ago
[deleted]
GuB-42 1y ago
I think the myth about exactness is that you can't use strict equality with floating point numbers because they are somehow fuzzy. They are not. Some operations involve rounding, notably conversion from decimal, but the (rounded) result is an exact number that can be stored and recovered without loss of precision and equality will work as expected. Converting from floating point binary to decimal is exact though, given enough digits (can be >1000 for very small or very large numbers).
- Joker_vD 1y ago
  > you can't use strict equality with floating point numbers because they are somehow fuzzy. They are not. They are though. All arithmetic operations involve rounding, so e.g. (7.0 / 1234 + 0.5) * 1234 is not equal to 7.0 + 617 (it differs in 1 ULP). On the other hand, (9.0 / 1234 + 0.5) * 1234 is equal to 9.0 + 617, so the end result is sometimes exact and sometimes is not. How can you know beforehand which one is the case in your specific case? Generally, you can't, any arithmetic operation can potentially give you 1 ULP of error, and it can (and likely, will) slowly accumulate.

AndriyKunitsyn 1y ago

Here's one that's not a myth: IEEE-754 floats are the only "primitive types" that allow "a == a" to not be true. I.e., two floats that are _identical_ to each other (even when it's _the same_ variable, on the same memory address) can be not _equal_ to each other, specifically if it's NaN. This is dictated by IEEE-754, and this is true for all programming languages I know, and to this day, this makes zero sense to me, but apparently this is useful for some reason.

zahlman 1y ago
>but apparently this is useful for some reason. It lets you easily test whether a value is NaN without needing a library function call. (Even if the language wanted to provide a NaN literal, there's more than one NaN value, so that wouldn't actually work!)
- AndriyKunitsyn 1y ago
  This test is better with a function call, because it clearly shows the intent. Perf-wise, this function call will be inlined by any decent compiler. https://godbolt.org/z/chhM85vP3 https://godbolt.org/z/chhM85vP3 Breaking a basic programming expectation for a specific class of values of a specific type is not a good thing, in my opinion.

analog31 1y ago

>>> A beginner would use them, trusting they are infinitely capable and precise, which can lead to problems. An intermediate programmer knows that they have some limitations, and so by using some good practices the problems can be avoided. Amusingly, pocket calculators became affordable while I was in high school, and anybody who was interested in math learned the foibles of floating point almost instantly. Now it's an advanced topic. A difference may have been that our calculators had very few significant digits, so the errors introduced by successive calculations became noticeable more quickly. Also, you had to think about what you were doing because, even if you trusted the calculator, you didn't trust your fingers.

pclmulqdq 1y ago
Modern pocket calculators now use some very fancy math that makes them have much less error than if they used floating point. Many of them will use arbitrary-precision arithmetic at minimum, and there are some fancier schemes.

fiforpg 1y ago

In the context of double precision the article says > the largest integer value that can be represented exactly is 2^53 — I am confused as to why it not 2^52, given that there are 52 bits of mantissa, so relative accuracy is 2^-52, which translates to absolute accuracy larger than 1 after 2^52. Compare this to the table there saying "Next value after 1 = 1 + 2^-52".

pclmulqdq 1y ago
There's an implied one bit, so you actually have a 53 bit significand (and 53-bit precision) given only a 52 bit mantissa.
- fiforpg 1y ago
  Right, I did realize after posting that close to numbers of the form 1{hidden bit} + (1-2^-52){mantissa with all ones} the relative accuracy — corresponding to the absolute accuracy of a single bit in mantissa — is about 2^-53. The hidden bit is easy to forget about...
layer8 1y ago
https://stackoverflow.com/questions/1848700/biggest-integer-that-can-be-stored-in-a-double https://stackoverflow.com/questions/1848700/biggest-integer-...

layer8 1y ago

Archive of the defunct tweet link: https://web.archive.org/web/20210608061416/https://twitter.com/j_bikker/status/1372273107197698054 https://web.archive.org/web/20210608061416/https://twitter.c...

zX41ZdbW 1y ago

This is a great article! A few more resources on this topic: https://randomascii.wordpress.com/category/floating-point/ https://randomascii.wordpress.com/category/floating-point/ - a series of articles about floating point, with deeper details. https://float.exposed/ https://float.exposed/ - explorer of floating point values by Bartosz Ciechanowski.

zX41ZdbW 1y ago
Fun facts about floating point, from my practice: Obvious: An array containing NaNs will almost certainly be sorted incorrectly with std::sort. Non-obvious: sorting an array containing NaNs with std::sort from libstdc++ (gcc's implementation) could lead to a buffer overflow. Non-obvious: using MMX instructions without clearing the FPU state leads to incorrect results of floating-point instructions in other parts of the code base: https://github.com/simdjson/simdjson/issues/169 https://github.com/simdjson/simdjson/issues/169
extraduder_ire 1y ago
I love that float.exposed website. Really nails home a point I saw made a few years ago about how there's only a few billion possible floats, so you can often test them all.

vismit2000 1y ago

A famous old interview question regarding floating points: https://stackoverflow.com/questions/6699066/in-which-order-should-floats-be-added-to-get-the-most-precise-result https://stackoverflow.com/questions/6699066/in-which-order-s... This awesome article drives home the point very succinctly: https://fabiensanglard.net/floating_point_visually_explained/ https://fabiensanglard.net/floating_point_visually_explained...