4 ms·
I think you might be off by one for the "correct" result (i.e. 2^-55), which is what fsum arrives at. >>> 0.3 - 0.2 - 0.1 -2.7755575615628914e-17 >
by vitus 5y ago
I think you might be off by one for the "correct" result (i.e. 2^-55), which is what fsum arrives at.
>>> 0.3 - 0.2 - 0.1
-2.7755575615628914e-17
>>> (0.3 - 0.2 - 0.1) / 3
-9.25185853854297e-18
But either way, the fundamental problem in this case (as you noted) is actually due to loss of precision from numeric representation.
- MauranKilom 5y agoHeh, I reasoned about the exponent by hand and got -55 but then wasn't sure whether it was correct so I searched for discussions of 0.3 - 0.2 - 0.1 and took the value they quoted (2^-54). That being an off-by-one error does resolve my question.