3 ms·
Exactly! Scale by a power of two and most of these surprises will go away. 0.1 is particularly problematic because it happens not to be exactly representable
by mturmon 14y ago
Exactly! Scale by a power of two and most of these surprises will go away.
0.1 is particularly problematic because it happens not to be exactly representable as an IEEE FP number, or even as a finite-length binary sequence (http://www.wolframalpha.com/input/?i=0.1+to+binary http://www.wolframalpha.com/input/?i=0.1+to+binary). So taking exact numbers and multiplying them by this scale factor will tend to produce inexact numbers, which when truncated, will cause surprises.
Even scaling by something closer to a power of two would be better (1/16 + 1/32 = 0.09375).