4 ms·
Most people don't realize they have them. For instance in most BI/ETL tools the expression below returns false: 0.1 + 0.2 = 0.3 In Excel and Google Sheets
by dgudkov 3y ago
Most people don't realize they have them. For instance in most BI/ETL tools the expression below returns false:
0.1 + 0.2 = 0.3
In Excel and Google Sheets, this returns false:
2.03 - 0.03 - 2 = 0
- zokier 3y agoIn both of those cases the problem is not problem with floating point arithmetic but the parsing of floating point literals, or rather how systems take input and then substitute it with something different quietly behind the scenes. None of 0.1, 0.2, 0.3 are true (binary) fp values. Nobody would be surprised that 0.1000000000000000056 + 0.2000000000000000111 != 0.2999999999999999889
- dgudkov 3y agoYou won't believe it, but Excel users are actually surprised. As do users of many other applications.
- tialaramex 3y agoSomebody pointed me at "Towards an API for the Real Numbers" which explains why these calculations work how you expect in the Android default Calculator. It's really nice, as they explain you can't drop this in instead of the floating point arithmetic in a serious language because the performance isn't what you want. However in human terms, for a product like the calculator - it's easily fine.
- Dylan16807 3y agoThat's a very different kind of accuracy, though. This tool is about improving the number of digits that are correct. Trying to use exact equality is going to fail on most equations even if you have millions of correct digits.