3 ms·

This is definitely not just a matter of using double where something longer would have been better. 999999999999999 and 999999999999997 are both exactly represe

by gjm11 17y ago

This is definitely not just a matter of using double where something longer would have been better. 999999999999999 and 999999999999997 are both exactly representable as 64-bit IEEE floating-point numbers (i.e., doubles). They're not using single-precision floats either; with one 9 fewer in each number, everything's fine, but those numbers are way out of range for 32-bit IEEE floats. Doesn't look like fixed-precision decimals, either: reduce the second number by 1 and you correctly get a difference of 3.

gjm11 17y ago
Tweaking that second number, I find that when the difference is 2 or less we get 0, but otherwise we get the right answer. More evidence that it's not simply a FP imprecision problem. Further, if I approximately halve them both (just change the initial 9 to a 4) then a difference of 1 or less produces 0 and other results are correct. I think what's going on is more likely this: whoever implemented this wanted to avoid giving nonzero answers to calculations whose correct answer is zero, and therefore put in some sort of deliberate truncation somewhere. (It's not clear to me exactly what -- maybe every addition or subtraction that produces an answer much much smaller than its inputs gets clamped to 0, or something.) I don't think the truncation is happening purely at the output stage: if I as it to do the original calculation and multiply the result by 100, e.g., I still get 0. (I had to write it like that because otherwise HN's asterisks-mean-italics hack scrambles it. Even though the supposedly matching asterisk is in a different paragraph. PG, if you're reading this, I'd love it if you were to change that behaviour.) In other words, I think we're seeing an unfortunate effect of the same feature that makes it say that cos(2*atan(1)) is 0 rather than reporting it as about 6.123 x 10^-17 (the actual result in IEEE doubles).

by gjm11 17y ago

gjm11 17y ago

Tweaking that second number, I find that when the difference is 2 or less we get 0, but otherwise we get the right answer. More evidence that it's not simply a FP imprecision problem. Further, if I approximately halve them both (just change the initial 9 to a 4) then a difference of 1 or less produces 0 and other results are correct. I think what's going on is more likely this: whoever implemented this wanted to avoid giving nonzero answers to calculations whose correct answer is zero, and therefore put in some sort of deliberate truncation somewhere. (It's not clear to me exactly what -- maybe every addition or subtraction that produces an answer much much smaller than its inputs gets clamped to 0, or something.) I don't think the truncation is happening purely at the output stage: if I as it to do the original calculation and multiply the result by 100, e.g., I still get 0. (I had to write it like that because otherwise HN's asterisks-mean-italics hack scrambles it. Even though the supposedly matching asterisk is in a different paragraph. PG, if you're reading this, I'd love it if you were to change that behaviour.) In other words, I think we're seeing an unfortunate effect of the same feature that makes it say that cos(2*atan(1)) is 0 rather than reporting it as about 6.123 x 10^-17 (the actual result in IEEE doubles).