3 ms·
I could understand 0, but how does it get 2?
by sp332 8y ago
I could understand 0, but how does it get 2?
- twtw 8y agoThe nearest doubles to each of these two decimal constants end up being roughly 2 apart. Whereas for fp32 both decimal constants are stored as the same float.
- Dylan16807 8y agox999 rounds up when stored, x998 stays the same.
- theoh 8y agoThese integers are so large that they cannot be precisely represented by 32 bit or 64 bit floats. So there's a rounding effect. https://stackoverflow.com/a/1848953 https://stackoverflow.com/a/1848953
- your-nanny 8y agobecause 2 is the interval of precision at that scale. In floating point loss of precision scales with magnitude.
- codeflo 8y agoFP numbers are (roughly) stored in the form m×2^e (m = mantissa, e = exponent). When numbers can't be represented exactly, m is rounded. My guess is that these numbers end up being encoded as 4999999999999999×2 and 4999999999999999.5×2, where the latter is rounded up to 5000000000000000×2.
- tomxor 8y agoBecause 9999999999999999 is rounded to 10000000000000000 before any math even happens. Precision != order of magnitude.