3 ms·
Would it have helped to use, for example, 0.0625 or 0.125 as the scaling factor, instead of 0.1?
by 0x0 14y ago
Would it have helped to use, for example, 0.0625 or 0.125 as the scaling factor, instead of 0.1?
- mturmon 14y agoExactly! Scale by a power of two and most of these surprises will go away. 0.1 is particularly problematic because it happens not to be exactly representable as an IEEE FP number, or even as a finite-length binary sequence (http://www.wolframalpha.com/input/?i=0.1+to+binary http://www.wolframalpha.com/input/?i=0.1+to+binary). So taking exact numbers and multiplying them by this scale factor will tend to produce inexact numbers, which when truncated, will cause surprises. Even scaling by something closer to a power of two would be better (1/16 + 1/32 = 0.09375).