3 ms·
Except that neither double nor single precision IEEE floats behave the same way as whatever numbers Google is using.
by dreish 17y ago
Except that neither double nor single precision IEEE floats behave the same way as whatever numbers Google is using.
- swolchok 17y agoThe algorithm probably first decides how to do the math and then does the math. EDIT: single-precision does behave this way: -bash-3.2$ cat precision.c #include <stdio.h> int main() { float f1 = 999999999999999; float f2 = 999999999999997; printf("%f\n", f1 - f2); return 0; } -bash-3.2$ gcc precision.c -bash-3.2$ ./a.out 0.000000 I am saying that if it's not consistent with single-precision across multiple inputs, there's probably code that looks at the numbers involved and decides what kind of numeric type to use to do the calculations. In this case, single-precision floating-point happens to be a bad choice. Second edit: replacing the 7 with a 6 still prints 0. The parent is right: something else is going on here. I like the theory espoused elsewhere in this discussion that it's a result of some truncation to avoid returning nonzero when the result should be zero.
- gjm11 17y agoWhat exactly do you mean, and how does it make "using floating point arithmetic with insufficient precision" a viable explanation for this, given that none of the usual floating-point formats behave in this way?