12 ms·

The Floating-Point Guide (2010)

86 points by arunc 7y ago

brazzy 7y ago
Discussion when I created the site and first submitted it to HN: https://news.ycombinator.com/item?id=1257610 https://news.ycombinator.com/item?id=1257610
emptybits 7y ago
FWIW and for all its other shortcomings, Raku (nee Perl 6) handles common arithmetic in a manner that wouldn't surprise a mathematician. Or non-mathematician, for that matter. Witness: $ python3 -c 'print(0.3 == 0.1 + 0.2)' False $ ruby -e 'puts 0.3 == 0.1 + 0.2' false $ perl6 -e 'say 0.3 == 0.1 + 0.2' True
- SamReidHughes 7y ago
  For your amusement, Go: fmt.Print(0.3 == 0.1 + 0.2) // => true https://repl.it/repls/SubtleQuarrelsomeMaintenance https://repl.it/repls/SubtleQuarrelsomeMaintenance
  - tomtomtom777 7y ago
    Interesting. I thought Go used standard IEEE-754 floats. Why does it have this result?
    - SamReidHughes 7y ago
      They evaluate constant expressions exactly at compile time, then convert to float.
      - WalterBright 7y ago
        That works great until people complain that it works differently at compile time, and they have a point.
        SamReidHughes 7y ago
        Yeah. On the other hand, they won’t get any surprises when cross-compiling. Edit: Okay, maybe Go devs saved themselves the work of having floating point emulation. Edit: No, they are folding both constant expressions exactly and actual floating point operations, going by assembly output.
        gpderetta 7y ago
        GCC at least evaluates constant FP expressions using a model of the target machine FP unit, so that compile time and runtime expressions behave identically.
        tomtomtom777 7y ago
        Go uses IEEE-754. The behaviour of these may be odd for those new to the quirks of the float, but I thought the behaviour is nowadays standardized and reproducable across platforms.
        tomtomtom777 7y ago
        > No, they are folding both constant expressions exactly and actual floating point operations, going by assembly output. Yes. The behaviour is quite odd and unexpected. It seems they have different rules for when to apply the constant version of floating math than what constitutes actual compile time evaluation. const a float64 = 0.1; const b float64 = 0.2; const x float64 = 0.1 + 0.2; // Const evaluation, "rational" math const y float64 = a + b; // Const evaluation, IEEE-754 math fmt.Println(x == y) // false ??!? This seems quite a horrible approach. EDIT: Or how about this one: const x float64 = 0; fmt.Println(x + 0.1 + 0.2 == 0.1 + 0.2 + x ); // false! fmt.Println(x + 0.1 + 0.2 == x + (0.1 + 0.2) ); // false!
        SamReidHughes 7y ago
        The literal folding is specified language behavior, the other is just an optimization.
  - jancsika 7y ago
    Raku: Here's a trivial example that shows a sophisticated approach to getting you the precise numbers you want for all your numeric needs. Go: Here's a trivial example.
- enriquto 7y ago
  As a mathematician, I am actually very surprised by perl's 6 behavior here. What the hell is going on? Does it use fixed point arithmetic or what?
  - SamReidHughes 7y ago
    They have this totally irrational idea of having "rational" numbers. https://docs.perl6.org/language/numerics#Rational https://docs.perl6.org/language/numerics#Rational By irrational I am not joking, these sorts of number systems are complicated and glitchy, in this case with program-destroying magic like autoconversion to floating point when you hit a 2^64 denominator, and, don't click this URL, please, just read it: https://docs.perl6.org/language/numerics#Zero-denominator_rationals https://docs.perl6.org/language/numerics#Zero-denominator_ra...
    - lizmat 7y ago
      > totally irrational idea Could you expand on why it is a totally irrational idea? Seems pretty rational to me. But then I'm not really a mathematician. > autoconversion to floating point when you hit a 2^64 denominator Please note that this is after normalization. And it has come from practicality: having the denominator also be an BigInt, slows down the use of the Rational number significantly. However, if you do need that type of precision, then you can by using infectious FatRats: any expression with a FatRat, will result in a FatRat (unless explicitly coerced to something else, of course). With regards to the Zero-denominator Rats: they are just special cases just as IEEE has special cases for -Inf, Inf and NaN. So what's the problem there?
      - SamReidHughes 7y ago
        It adds gratuitous corner cases to the language.
  - lizmat 7y ago
    FWIW, Perl 6 prefers to be called Raku now.
  - 7y ago
- mansoor_ 7y ago
  Arguable that this is a good thing.
IEEE754 7y ago
FWIW, their "basic answer" page is the simplest i've seen that neither lies or omits critical factors in the 0.1 + 0.2 problem. It's probably a good starting point that induces some lingering questions tempting you to find out more. If you want a thorough understanding you will want to look at representation-error, rounding-error, error-propagation, why they exist and how they interact. The interplay between those three forms of numerical error in floating point numbers will also allow you to more easily see the world of limitations of fp beyond 0.1 + 0.2 for yourself.
- ken 7y ago
  I understand the "problem" from the hardware perspective, but I still don't accept their "basic answer" as reasonable. > It’s not stupid, just different. Over the past 50 years, my computer has adapted to how humans normally operate in nearly every other way. Why do they continue to use this system which produces results different from what any normal person expects? > Computers use binary numbers because they’re faster at dealing with those Computers are faster at dealing with all-caps ASCII, too, but we've accepted here that micro-optimization is less important than doing what people want. Most of the languages I use have even moved past fixnums. Why have we not improved real arithmetic since 1985?
  - gpderetta 7y ago
    because FLOPS do not grow on trees.
  - msla 7y ago
    > Over the past 50 years, my computer has adapted to how humans normally operate in nearly every other way. Why do they continue to use this system which produces results different from what any normal person expects? Lisps and Lisp-derived languages, like Scheme, have had a proper numerical tower, including rationals, for decades now. Using reals is optional, but using rationals and everything else imposes an efficiency cost, so people make their decision. Implementing rationals in hardware would not necessarily make them more efficient; that is, if you think having rational support in hardware would help, you have to make the case. It isn't an automatic win: https://yosefk.com/blog/its-done-in-hardware-so-its-cheap.html https://yosefk.com/blog/its-done-in-hardware-so-its-cheap.ht...
userbinator 7y ago
Given the name I thought the site would have a GUI for playing around with floating-point numbers, like this one: https://www.exploringbinary.com/floating-point-converter/ https://www.exploringbinary.com/floating-point-converter/ The "On Using Integers" page seems very narrowly applicable because it only discusses fixed-point in the context of currency; in a lot of other applications, notably DSP and other multimedia, fixed-point is the norm.
- okaleniuk 7y ago
  I have some playable GUI too: https://wordsandbuttons.online/yet_another_floating_point_tutorial.html https://wordsandbuttons.online/yet_another_floating_point_tu...
- brazzy 7y ago
  The page is about using fixed point types to avoid binary rounding, which is the underlying issue the whole website is about. AFAIK in DSPs you use fixed point for a completely different reason, namely because it's simpler and thus more efficient to implement in silicon.
roland35 7y ago
You have to be very careful using floating point on embedded processors not only for the reasons in this website, but for speed too! Unless you have a separate floating-point core (ARM Cortex M4s have one, M3s do not) it can take 1,000's of clock cycles to perform any floating point math like addition, multiplication, etc. Even with a floating point processor you need to be careful when switching into and out of interrupts! The floating point core has it's own set of registers you need to keep track of. Doubles are even worse! Basically it is almost always better to use integers when you can, and be aware of what types you are using (always important in C anyways).
- petermcneeley 7y ago
  Something seems off about your numbers given that you can emulate fp with integer ops in much less than 1000 instructions.
  - gpderetta 7y ago
    probably because on cpu without fp units, fp instructions cause invalid instruction exceptions to be raised that need to handled by the kernel, and that can be very expensive (that used to be a thing on x86 in the past).
    - Symmetry 7y ago
      Kernel? What kernel? You probably aren't using a separate kernel with an M3, though you might have an RTOS. You'll, as a rule, have the compiler putting the floating point emulations in directly. I think the performance hit is only around a factor of 100 or so for an M3 though.
  - brandmeyer 7y ago
    It is way off. GCC has had optimized assembly routines for performing software floating point on ARM that run in dozens of cycles. IIRC, the license is liberal enough that LLVM could copy them by-value into their compiler support libraries without difficulty, so I would just assume that they were until proven otherwise.
giu 7y ago
A very useful resource that I've used quite a few times in the past for comparing floats: https://floating-point-gui.de/errors/comparison/ https://floating-point-gui.de/errors/comparison/. The latter also provides the following link to unit tests that cover a lot of edge cases: https://floating-point-gui.de/errors/NearlyEqualsTest.java https://floating-point-gui.de/errors/NearlyEqualsTest.java
gnufx 7y ago
Wot, no Goldberg? E.g. https://dl.acm.org/citation.cfm?id=103163 https://dl.acm.org/citation.cfm?id=103163 or https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.h...
- brazzy 7y ago
  It's in the links of course, but I wrote the site specifically because I saw it being used too often as a kneejerk response to basic questions, where it's not helpful at all.
  - simonbyrne 7y ago
    Thank you! The Goldberg article is a terrible way to learn about floating point, and the frequency with which it is referred to on StackOverflow is really disheartening.
    - fargle 7y ago
      Why do you say that? It is an extremely good paper and provides a practical and high-level overview in an accessible (details to follow) format. If it's still just too technical, then a) know there is are "subtleties" and b) use double precision c) it's hard because... reasons.
      - simonbyrne 7y ago
        I disagree about practical and accessible: - its exposition is complicated (trying to prove everything in a general base makes it difficult to understand) - it's woefully out of date (lack of guard digits haven't been an issue for at least 25 years, extended precision hasn't been an issue for the past 10 or so, and most languages now default to having fairly strict floating point semantics) - it gets bogged down in irrelevant minutiae (rounding modes and exception flags, while available in modern hardware, aren't really supported by any modern languages/compilers) - it doesn't really provide any practical advice (it barely mentions binary-decimal conversion, it jumps to doubling precision and Kahan summation without suggesting any intermediate steps such as sorted or pairwise summation). But my biggest complaint is the frequency with which users are referred to it on StackOverflow as if (1) it is a good way to learn about floating point concepts, and (2) anyone using floating point numbers should be expected to understand it all.
fargle 7y ago
This article is terrible. Referring to the Goldberg paper as "long article with loltz of formulas" in a throwaway blog post shorter than most of these comments is telling. The "absurd" result that [1+2 != 3] * (0.1) isn't even the problem with floating point. It's an even simpler misconception that "even" vs. "repeating" fractions are the same for base 10 vs. base 2 fractions. So we can understand that 0.33+0.33+0.33 != 1 in decimal due to rounding. So when 0.1 (decimal) which is a truncated repeating "binary decimal" fraction isn't exact we loose our shit. But why is base 10 good and base 2 bad? Why not base 57? Clearly for any base, a truncated decimal-like fraction will be inexact for a lot of numbers. Rounding errors will add up, but this is really a binary vs. decimal issue. More subtle are errors that grow simply because of issues specific to floating point. For example re-normalization when numbers of very different magnitudes are added or subtracted which eventually causes precision loss. This is neither hard to understand, nor the only complication with floating point, nor is anything wrong, or bad, nor should anything be fixed in IEEE-754. This is just stuff that numeric programmers MUST understand, at least at the rudimentary level.
- brazzy 7y ago
  It's telling how you call the website "terrible", and then proceed to do exactly what it does: give a basic, factual explanation why floating point math does not behave as people naively expect. No, it is not hard to understand, but it is not obvious, and in fact counterintuitive to most people. Which is why there is a need for a website like this for people to refer to intead of the Goldberg paper which is great if you actually need to do error propagation analysis on your numeric algorithm but which does not give this basic explanation and is therefore not suitable to be cited to people who are suprised that 0.1 + 0.2 != 0.3 So please explain to me how referring to the Goldberg paper as "long article with lols of formulas" is anything but factually correct and a good reason to provide an alternative?
  - fargle 7y ago
    OK, let's try again. I gave a correct summary of what I've learned from Goldberg and elsewhere. In a short comment. This article is "terrible" because it gives a single bad example and derides a very good reference implying it's too complicated. And steals it's title. Every programmer that uses floating point should read Goldberg. That blog post is garbage. Post the real source, or an informative article, not junk.

86 points by arunc 7y ago

Discussion when I created the site and first submitted it to HN: https://news.ycombinator.com/item?id=1257610 https://news.ycombinator.com/item?id=1257610

emptybits 7y ago

FWIW and for all its other shortcomings, Raku (nee Perl 6) handles common arithmetic in a manner that wouldn't surprise a mathematician. Or non-mathematician, for that matter. Witness: $ python3 -c 'print(0.3 == 0.1 + 0.2)' False $ ruby -e 'puts 0.3 == 0.1 + 0.2' false $ perl6 -e 'say 0.3 == 0.1 + 0.2' True

SamReidHughes 7y ago
For your amusement, Go: fmt.Print(0.3 == 0.1 + 0.2) // => true https://repl.it/repls/SubtleQuarrelsomeMaintenance https://repl.it/repls/SubtleQuarrelsomeMaintenance
- tomtomtom777 7y ago
  Interesting. I thought Go used standard IEEE-754 floats. Why does it have this result?
  - SamReidHughes 7y ago
    They evaluate constant expressions exactly at compile time, then convert to float.
    - WalterBright 7y ago
      That works great until people complain that it works differently at compile time, and they have a point.
      - SamReidHughes 7y ago
        Yeah. On the other hand, they won’t get any surprises when cross-compiling. Edit: Okay, maybe Go devs saved themselves the work of having floating point emulation. Edit: No, they are folding both constant expressions exactly and actual floating point operations, going by assembly output.
        gpderetta 7y ago
        GCC at least evaluates constant FP expressions using a model of the target machine FP unit, so that compile time and runtime expressions behave identically.
        tomtomtom777 7y ago
        Go uses IEEE-754. The behaviour of these may be odd for those new to the quirks of the float, but I thought the behaviour is nowadays standardized and reproducable across platforms.
        tomtomtom777 7y ago
        > No, they are folding both constant expressions exactly and actual floating point operations, going by assembly output. Yes. The behaviour is quite odd and unexpected. It seems they have different rules for when to apply the constant version of floating math than what constitutes actual compile time evaluation. const a float64 = 0.1; const b float64 = 0.2; const x float64 = 0.1 + 0.2; // Const evaluation, "rational" math const y float64 = a + b; // Const evaluation, IEEE-754 math fmt.Println(x == y) // false ??!? This seems quite a horrible approach. EDIT: Or how about this one: const x float64 = 0; fmt.Println(x + 0.1 + 0.2 == 0.1 + 0.2 + x ); // false! fmt.Println(x + 0.1 + 0.2 == x + (0.1 + 0.2) ); // false!
        SamReidHughes 7y ago
        The literal folding is specified language behavior, the other is just an optimization.
- jancsika 7y ago
  Raku: Here's a trivial example that shows a sophisticated approach to getting you the precise numbers you want for all your numeric needs. Go: Here's a trivial example.
enriquto 7y ago
As a mathematician, I am actually very surprised by perl's 6 behavior here. What the hell is going on? Does it use fixed point arithmetic or what?
- SamReidHughes 7y ago
  They have this totally irrational idea of having "rational" numbers. https://docs.perl6.org/language/numerics#Rational https://docs.perl6.org/language/numerics#Rational By irrational I am not joking, these sorts of number systems are complicated and glitchy, in this case with program-destroying magic like autoconversion to floating point when you hit a 2^64 denominator, and, don't click this URL, please, just read it: https://docs.perl6.org/language/numerics#Zero-denominator_rationals https://docs.perl6.org/language/numerics#Zero-denominator_ra...
  - lizmat 7y ago
    > totally irrational idea Could you expand on why it is a totally irrational idea? Seems pretty rational to me. But then I'm not really a mathematician. > autoconversion to floating point when you hit a 2^64 denominator Please note that this is after normalization. And it has come from practicality: having the denominator also be an BigInt, slows down the use of the Rational number significantly. However, if you do need that type of precision, then you can by using infectious FatRats: any expression with a FatRat, will result in a FatRat (unless explicitly coerced to something else, of course). With regards to the Zero-denominator Rats: they are just special cases just as IEEE has special cases for -Inf, Inf and NaN. So what's the problem there?
    - SamReidHughes 7y ago
      It adds gratuitous corner cases to the language.
- lizmat 7y ago
  FWIW, Perl 6 prefers to be called Raku now.
- 7y ago
mansoor_ 7y ago
Arguable that this is a good thing.

IEEE754 7y ago

FWIW, their "basic answer" page is the simplest i've seen that neither lies or omits critical factors in the 0.1 + 0.2 problem. It's probably a good starting point that induces some lingering questions tempting you to find out more. If you want a thorough understanding you will want to look at representation-error, rounding-error, error-propagation, why they exist and how they interact. The interplay between those three forms of numerical error in floating point numbers will also allow you to more easily see the world of limitations of fp beyond 0.1 + 0.2 for yourself.

ken 7y ago
I understand the "problem" from the hardware perspective, but I still don't accept their "basic answer" as reasonable. > It’s not stupid, just different. Over the past 50 years, my computer has adapted to how humans normally operate in nearly every other way. Why do they continue to use this system which produces results different from what any normal person expects? > Computers use binary numbers because they’re faster at dealing with those Computers are faster at dealing with all-caps ASCII, too, but we've accepted here that micro-optimization is less important than doing what people want. Most of the languages I use have even moved past fixnums. Why have we not improved real arithmetic since 1985?
- gpderetta 7y ago
  because FLOPS do not grow on trees.
- msla 7y ago
  > Over the past 50 years, my computer has adapted to how humans normally operate in nearly every other way. Why do they continue to use this system which produces results different from what any normal person expects? Lisps and Lisp-derived languages, like Scheme, have had a proper numerical tower, including rationals, for decades now. Using reals is optional, but using rationals and everything else imposes an efficiency cost, so people make their decision. Implementing rationals in hardware would not necessarily make them more efficient; that is, if you think having rational support in hardware would help, you have to make the case. It isn't an automatic win: https://yosefk.com/blog/its-done-in-hardware-so-its-cheap.html https://yosefk.com/blog/its-done-in-hardware-so-its-cheap.ht...

userbinator 7y ago

Given the name I thought the site would have a GUI for playing around with floating-point numbers, like this one: https://www.exploringbinary.com/floating-point-converter/ https://www.exploringbinary.com/floating-point-converter/ The "On Using Integers" page seems very narrowly applicable because it only discusses fixed-point in the context of currency; in a lot of other applications, notably DSP and other multimedia, fixed-point is the norm.

okaleniuk 7y ago
I have some playable GUI too: https://wordsandbuttons.online/yet_another_floating_point_tutorial.html https://wordsandbuttons.online/yet_another_floating_point_tu...
brazzy 7y ago
The page is about using fixed point types to avoid binary rounding, which is the underlying issue the whole website is about. AFAIK in DSPs you use fixed point for a completely different reason, namely because it's simpler and thus more efficient to implement in silicon.

roland35 7y ago

You have to be very careful using floating point on embedded processors not only for the reasons in this website, but for speed too! Unless you have a separate floating-point core (ARM Cortex M4s have one, M3s do not) it can take 1,000's of clock cycles to perform any floating point math like addition, multiplication, etc. Even with a floating point processor you need to be careful when switching into and out of interrupts! The floating point core has it's own set of registers you need to keep track of. Doubles are even worse! Basically it is almost always better to use integers when you can, and be aware of what types you are using (always important in C anyways).

petermcneeley 7y ago
Something seems off about your numbers given that you can emulate fp with integer ops in much less than 1000 instructions.
- gpderetta 7y ago
  probably because on cpu without fp units, fp instructions cause invalid instruction exceptions to be raised that need to handled by the kernel, and that can be very expensive (that used to be a thing on x86 in the past).
  - Symmetry 7y ago
    Kernel? What kernel? You probably aren't using a separate kernel with an M3, though you might have an RTOS. You'll, as a rule, have the compiler putting the floating point emulations in directly. I think the performance hit is only around a factor of 100 or so for an M3 though.
- brandmeyer 7y ago
  It is way off. GCC has had optimized assembly routines for performing software floating point on ARM that run in dozens of cycles. IIRC, the license is liberal enough that LLVM could copy them by-value into their compiler support libraries without difficulty, so I would just assume that they were until proven otherwise.

giu 7y ago

A very useful resource that I've used quite a few times in the past for comparing floats: https://floating-point-gui.de/errors/comparison/ https://floating-point-gui.de/errors/comparison/. The latter also provides the following link to unit tests that cover a lot of edge cases: https://floating-point-gui.de/errors/NearlyEqualsTest.java https://floating-point-gui.de/errors/NearlyEqualsTest.java

gnufx 7y ago

Wot, no Goldberg? E.g. https://dl.acm.org/citation.cfm?id=103163 https://dl.acm.org/citation.cfm?id=103163 or https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.h...

brazzy 7y ago
It's in the links of course, but I wrote the site specifically because I saw it being used too often as a kneejerk response to basic questions, where it's not helpful at all.
- simonbyrne 7y ago
  Thank you! The Goldberg article is a terrible way to learn about floating point, and the frequency with which it is referred to on StackOverflow is really disheartening.
  - fargle 7y ago
    Why do you say that? It is an extremely good paper and provides a practical and high-level overview in an accessible (details to follow) format. If it's still just too technical, then a) know there is are "subtleties" and b) use double precision c) it's hard because... reasons.
    - simonbyrne 7y ago
      I disagree about practical and accessible: - its exposition is complicated (trying to prove everything in a general base makes it difficult to understand) - it's woefully out of date (lack of guard digits haven't been an issue for at least 25 years, extended precision hasn't been an issue for the past 10 or so, and most languages now default to having fairly strict floating point semantics) - it gets bogged down in irrelevant minutiae (rounding modes and exception flags, while available in modern hardware, aren't really supported by any modern languages/compilers) - it doesn't really provide any practical advice (it barely mentions binary-decimal conversion, it jumps to doubling precision and Kahan summation without suggesting any intermediate steps such as sorted or pairwise summation). But my biggest complaint is the frequency with which users are referred to it on StackOverflow as if (1) it is a good way to learn about floating point concepts, and (2) anyone using floating point numbers should be expected to understand it all.

fargle 7y ago

This article is terrible. Referring to the Goldberg paper as "long article with loltz of formulas" in a throwaway blog post shorter than most of these comments is telling. The "absurd" result that [1+2 != 3] * (0.1) isn't even the problem with floating point. It's an even simpler misconception that "even" vs. "repeating" fractions are the same for base 10 vs. base 2 fractions. So we can understand that 0.33+0.33+0.33 != 1 in decimal due to rounding. So when 0.1 (decimal) which is a truncated repeating "binary decimal" fraction isn't exact we loose our shit. But why is base 10 good and base 2 bad? Why not base 57? Clearly for any base, a truncated decimal-like fraction will be inexact for a lot of numbers. Rounding errors will add up, but this is really a binary vs. decimal issue. More subtle are errors that grow simply because of issues specific to floating point. For example re-normalization when numbers of very different magnitudes are added or subtracted which eventually causes precision loss. This is neither hard to understand, nor the only complication with floating point, nor is anything wrong, or bad, nor should anything be fixed in IEEE-754. This is just stuff that numeric programmers MUST understand, at least at the rudimentary level.

brazzy 7y ago
It's telling how you call the website "terrible", and then proceed to do exactly what it does: give a basic, factual explanation why floating point math does not behave as people naively expect. No, it is not hard to understand, but it is not obvious, and in fact counterintuitive to most people. Which is why there is a need for a website like this for people to refer to intead of the Goldberg paper which is great if you actually need to do error propagation analysis on your numeric algorithm but which does not give this basic explanation and is therefore not suitable to be cited to people who are suprised that 0.1 + 0.2 != 0.3 So please explain to me how referring to the Goldberg paper as "long article with lols of formulas" is anything but factually correct and a good reason to provide an alternative?
- fargle 7y ago
  OK, let's try again. I gave a correct summary of what I've learned from Goldberg and elsewhere. In a short comment. This article is "terrible" because it gives a single bad example and derides a very good reference implying it's too complicated. And steals it's title. Every programmer that uses floating point should read Goldberg. That blog post is garbage. Post the real source, or an informative article, not junk.