22 ms·
Comparing Floating-Point Numbers Is Tricky (2017)
- magicalhippo 4y agoFloating-point numbers are tricky[1]. One point which often bites people which isn't highlighted in this article, is that due to having an explicit sign bit 0 and -0 have different encodings, but in most cases you'll want them to compare equal. So just comparing the bits won't do. IIRC even the .Net CLR had this issue way back in the 1.1 days. For some problems, it can often make sense to switch to arbitrary precision numbers, at least for some parts of the calculations. [1]: https://floating-point-gui.de/ https://floating-point-gui.de/
- maweki 4y agoYou mean back in the 1.099999904632568359375 days?
- arunc 4y agoProbably back in 1.100000004632568359375 days. Wait, when did HN become reddit?!
- Agingcoder 4y agoI do enjoy the occasional joke on hn - for reasons I don't understand they seem to be frowned upon.
- vbezhenar 4y ago> One point which often bites people which isn't highlighted in this article, is that due to having an explicit sign bit 0 and -0 have different encodings, but in most cases you'll want them to compare equal. So just comparing the bits won't do. How does JS do it? (0 / -1) === (0 / 1) is true.
- TAForObvReasons 4y agoconst n = 0/-1, p = 0/1; n === p; // true 1/n === 1/p; // false 1/n === -Infinity; // true 1/p === Infinity; // true Infinity === -Infinity; // false
- IshKebab 4y agoAnd because of that `x + 0` is not always equal to `x`.
- danybittel 4y agoWhat helped me better understand floats, was playing around with them in their bit representation. For example, if you go to https://www.h-schmidt.net/FloatConverter/IEEE754.html https://www.h-schmidt.net/FloatConverter/IEEE754.html and enter 1.0. Then use +1 and -1 to go through ULPs.
- stabbles 4y agoAnother nice tool is the Julia repl: julia> 1.4 + eps(1.4) === nextfloat(1.4) true julia> bitstring(1.4) "0011111111110110011001100110011001100110011001100110011001100110" julia> bitstring(nextfloat(1.4)) "0011111111110110011001100110011001100110011001100110011001100111"
- OscarCunningham 4y agoThe site I really like for exploring the binary representation of floats is https://float.exposed https://float.exposed.
- colinsane 4y agoprevious discussion: https://news.ycombinator.com/item?id=13998564 https://news.ycombinator.com/item?id=13998564 should have the 2017 tag. hi Matt :)
- OscarCunningham 4y agoCan someone give an example of a time you would actually want to compare floats for equality? Every time I've come across it there's always been a better way.
- jbay808 4y agoPerhaps when you are unit testing a math function on a known input, for a specific platform, and there is a specific value that you are expecting it to return; anything else is an error.
- VBprogrammer 4y agoI think the most common case, though it's wrong in a multitude of ways, is when you accidentally end up with a float representing a money amount. Things which are genuinely representable as a float value testing for equality is very rarely useful.
- enriquto 4y ago> Can someone give an example of a time you would actually want to compare floats for equality? When they are integers. By construction, floats provide exact arithmetic for small ints. It makes sense to compare those for equality. Also, when defining a ``sign'' function, you may want to treat the equality case separately, to avoid introducing biases when the input data is quantized.
- travisjungroth 4y agoI try to not worry about if my method of comparing floating point numbers is perfect. I figure close enough is close enough.
- throwamon 4y agoPyret has a pragmatic way of dealing with what they call "roughnums": https://www.pyret.org/docs/latest/equality.html https://www.pyret.org/docs/latest/equality.html
- jqpabc123 4y agoMy guess is that at least half of the cases where floats are applied could be better served using scaled integers. Real world practicalities limit the the need for extended precision in a lot of (if not most) cases. For example, most monetary transactions are settled to the nearest penny. Most time clocks are only accurate to the nearest second. Most milling machines are only accurate to a single digit fraction of a millimeter. Applying extended decimals to these may work --- but in engineering terms, this is really "false accuracy" that is unnecessary and can be counter productive.
- tialaramex 4y agoFloats are really nice for audio (PCM) which may not be intuitive. It's much nicer during mixing, editing and other production steps to work with floats and so essentially all modern software does that, even though the A2D conversion is integer and the end results will be integer data (e.g. CD audio is Linear PCM 16-bits at 44.1kHz)
- jqpabc123 4y agoIt's much nicer Said every developer who has ever applied floats where scaled integers would be just as effective. I attribute this to pocket calculators which have shaped most people's thinking. I've even had developers try to tell me it can't be done any other way --- until I show them. All that said, if floats are needed to interface with other software, you may not have much choice in the matter.
- tialaramex 4y agoNot for developers, it's much nicer for the users. There's a phrase "infinite headroom" which might make this a bit sharper. With a conventional analogue recording medium, the medium has physical properties which impact the precision with which it can record audio. For example maybe the reel-to-reel tape recorders in a studio can do about 12-bit precision. At the bottom, the precision is cut off by noise, mechanical noise in the equipment and eventually even quantum mechanical noise in our universe which is unavoidable. At the top it's cut off by practical engineering limits - the magnetic field strengths are limited. A good analogue engineer ensures that there's "headroom" where if the recording is a little louder than expected it doesn't bump into that upper limit, when Aretha hits that note 6dB louder than anticipated, you had 10dB of headroom so it's captured OK. But to do this you're sacrificing that precision at the other end of the scale, listeners can't hear her turning the lyrics sheet in the background of the song even though you'd have heard that if you were in the room. In digital integer space, we can have lots of precision, 14-bit happened early, 16-bit was soon common place, and today 24-bit is widely available (although you need to squint to see 24 actual bits, don't think about it too hard). But, the integers are brick walls at the edges of that range. If you set things up to use all your (say) 16-bits and it comes out hotter than expected, your data is trashed because it saturates † producing a harsh clipping. So you need to carefully set the headroom to give you as much space at the top as you need, but without wasting too much. Stressful. With floats you get infinite headroom. Conventionally PCM audio is represented as -1.0 to 1.0, but in float space if you're "too hot" by a factor of literally 1000, it's no problem at all, floating point numbers can go from -1000.0 to 1000.0 just fine, and when you're ready to actually listen to it, just multiply them by 0.001 (this is called "gain" in audio). † Or, if the programmer doesn't understand what they're doing, it wraps and sounds like complete nonsense. This is common on 1980s home computer audio setups.
- tomxor 4y agoI think this is the first time I've seen a simple, approachable article that makes correct use of machine epsilon (specifically treating it as a _relative_ error)... It's unfortunately very common for people to misguidedly treat it as an absolute error which makes no sense if you pause to think how it's derived and what it really means (the limit of the significand window, aka mantissa, i.e 2^-52 for a double)... that and the fact that if it's a constant it cannot possibly be an absolute error when the exponent is separate from the significand. There are libraries that have implemented conditions with macheps as an absolute, it's on my todo list to try convince the maintainers of such libraries that's it wrong, but it's a tricky subject to explain while also being massively underestimated by most, so I don't relish it - combined with the fact that checking against an arbitrarily very small absolute does happen to fix a lot of comparison errors - just not for the reason they think, and not reliably.
- conaclos 4y agoIt is why we decided [2] to exclude floats as map key in bare [1]. Go follows the standard. However, this creates an issue [3]. Indeed, NaN can map to an infinite number of values. These values cannot be accessed. +0 and -0 designates a unique key. JavaScript normalizes -0 to 0, and NaNs are normalized to a canonical NaN. This makes possible to use NaN as a key in a map. Other implementations could rely on binary comparison. This is a lot of pain to deal with. It is better to avoid float comparison or to restrict floats in some way. [1] https://baremessages.org https://baremessages.org [2] https://lists.sr.ht/~sircmpwn/public-inbox/%3CCAFFTG-a-Vci%2BkS_d_%3DuaX8kyxszqtB-79pcb5580Z9xw72V5kw%40mail.gmail.com%3E#%3C9dd16aad-ac82-7ea2-2bb0-3c856b4db850@elvinger.fr%3E https://lists.sr.ht/~sircmpwn/public-inbox/%3CCAFFTG-a-Vci%2... [3] https://github.com/golang/go/issues/20660 https://github.com/golang/go/issues/20660
- archi42 4y agoCame to say "when (not) to use floats is actually the tricky part", read the article, and still say it. Not that the article is wrong, but I think that's just one of the minor issues when people use floats. If engineers would only use floats once they properly understood the concepts, they'll naturally start to wonder how to properly do comparisons (especially for equality). To illustrate, here is some gold a friend recently showed to me: `float i = 1.F; while(i <= 10.f) { do_something_important(array[(int)i]); i+= 1.f; }`. We both died a little bit (though maybe only I did, because he sees stuff like that too often).
- andreygrehov 4y agoI'm not familiar with all the floating-point number gotchas, but would like to ask why an obvious solution is not used (or maybe it is?): convert both numbers to a string and compare the result? This might be a naive question, but still.
- Someone 4y agoConverting a IEEE float to a string is tricky, too, IMO more tricky. It would be a lot slower, too, and wouldn’t work. How do you define “compare the result”? The strings “0.0” and “-0.0” would have to be equal to each other, and the string “NaN” not equal to itself, “1000” larger than “0.999E3”, etc.
- constantcrying 4y agoIt is slow and doesn't really solve the problem. Representing the numbers in base 10 does not make it any clearer whether the numbers are equal.
- whycombinetor 4y agoAm I the only person who thinks of IEEE 754 as a completely well-defined data structure with well-defined arithmetic operations, where two numbers are equal iff their bits are equal? Tiny discrepancies don't randomly appear during float operations, and two different black box systems written from the same order of operations spec sheet (given proper compiler strict math settings) should be able to output the exact same float value given the same input.
- macrolocal 4y agoNot at all, eg. Knuth's TAOCP, Vol. 2, Section 4.2.2 A.
- adgjlsfhk1 4y agoThere are 2 problems with this. The first is that CPUs want to do out of order execution and vectorization, so not allowing reordering can come at a significant cost. The second problem is that different LIBMs and cpus will change the result of transcendental functions like exp.
- Someone 4y ago> where two numbers are equal iff their bits are equal? That’s not true. One of the features of IEEE floats is the existence of NaNs that aren’t (according to the standard) equal to themselves.
- tomxor 4y ago> Am I the only person who thinks of IEEE 754 as a completely well-defined data structure Not at all... but I think there are a large number of people who think they can imagine a better format and yet don't fully understand or appreciate much of the design of IEEE754, and they tend to be quite noisy. I think the lack of appreciation is ironically due to how damn good a format it is. It manages to hide the myriad deficiencies through such a good balance of compromises that it casts this illusion that perfect fp math is possible in a generalisable enough way. So on the occasions it surfaces the true reality, people assume it must be some kind of defect that is easily addressable with a new format; rather than an intentional decision to maximise correctness and utility by carefully choosing where and how to be incorrect. FP math is the most underestimated areas by programmers IMO, it's just taken for granted so much, which is probably a testament to how well IEEE754 works.
- bicsi 4y agoThe story is much worse than what is presented in the article, especially when talking about floating point errors that add (or rather multiply) up. More often than not, the error is relative wrt the greatest magnitudes in the intermediary calculations. In essence, if you subtract two floating point numbers, you’re kinda screwed, because you cannot ever handle with good precision cases like A - B, where A and B are big enough numbers. Not to mention more complicated operations like trig functions. In my opinion, one should avoid floating point as much as possible. And not only when testing for equality (all comparisons suffer from this). Or, of course, ignore FPEs and proceed at your own risk.
- constantcrying 4y agoThis is not the issue. Floating point numbers have problems when A and B differ greatly in magnitude, then A-B might easily be equal to A and so ((A-B)-A)LARGENUMBER can still be equal to zero, even if B was e.g. 1. This is not a problem with floats. This is an inherent* result of the design constraints. You can not fix it and there are no alternatives without this flaw which offer any of the same benefits as floats. Any operation between two floats is the floating point number which is closest to the result calculated as real numbers, that is the magic of floats. Floating point numbers are a near magical type, which for many applications are not only a good choice, but the only one which makes any sense. It is near impossible to imagine modern engineering without them.
- tomxor 4y agoIndeed. The only way to accommodate numbers where the difference in order of magnitude exceeds the type's significand length is to not combine them, keep them separate and operate on them separately, combining them only for a sample. One good common example is numerical integration. In any sufficient fine grained posteriori simulation, even with modest limits on position - delta velocity can be too small to preserve when adding to position, i.e when delta velocity is more than 2^52 smaller than position. Keeping a separate accumulator is the only way to handle this without arbitrarily increasing precision with software FP.
- everybodyknows 4y agoAlso posted today, What Every Computer Scientist Should Know About Floating-Point Arithmetic: https://news.ycombinator.com/item?id=33299829 https://news.ycombinator.com/item?id=33299829 Links to an excruciatingly thorough treatment of the subject by David Goldberg: https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.h...