3 ms·
How does Google get 0 out of that? Both my high school math teacher and IEEE 754 think that's wrong.
by aftbit 10y ago
How does Google get 0 out of that? Both my high school math teacher and IEEE 754 think that's wrong.
- Kubuxu 10y agoMaybe it is using single precision.
- PDoyle 10y ago9999999999999999.0 can't be represented as a double. Depending on the rounding mode, it could become 9999999999999998.0, so when you subtract them, they're zero. That happens when the rounding mode is round toward zero, or round down. Any other mode rounds it up to 1e+16, so you get an answer of 2.