3 ms·
Excel rounds doubles to 15 digits for display and comparison. The exact precision of doubles is something like 15.6 digits, those remaining 0.6 digits causing s
by slver 5y ago
Excel rounds doubles to 15 digits for display and comparison. The exact precision of doubles is something like 15.6 digits, those remaining 0.6 digits causing some of those examples floating (heh) around.
- okl 5y agoThat depends https://randomascii.wordpress.com/2012/03/08/float-precisionfrom-zero-to-100-digits-2/ https://randomascii.wordpress.com/2012/03/08/float-precision...
- slver 5y agoA lot of these edge cases are about theoretical concerns like "how many digits we need in decimal to represent an exact IEEE binary float". In practice a double is 15.6 digits precise, which Excel rounds to 15 to eliminate some weirdness. In their documentation they do cite their number type as 15 digit precision type. Ergo that's the semantic they've settled on.