4 ms·
Not sure how this is related, but: - if you get 0.2, the code is using rational numbers or a custom implementation - 0.199999...8 is the double precision IEEE
by qsort 6y ago
Not sure how this is related, but:
- if you get 0.2, the code is using rational numbers or a custom implementation
- 0.199999...8 is the double precision IEEE subtraction.
- 0.200000...2 is the single precision IEEE subtraction.
This looks like a typecasting/coercion issue. You can likely get all of the above in C++ or Java by using different explicit casts.
- dragonwriter 6y ago> if you get 0.2, the code is using rational numbers or a custom implementation IIRC, there are languages that use IEEE double precision but the way they handle default display presents this as 0.2 (basically, they use a display algorithm that displays the shortest decimal expression that has the same double precision representation.)
- CraigJPerry 6y ago0.2 could just be base 10 / decimal floating point rather than the base 2 single/double i think, It’s all ieee754 it’s just the base10 stuff came later
- vips7L 6y agoUsing doubles in Java you get 0.199999..8, using BigDecimal you get the correct 0.2. jshell> new BigDecimal("0.3").subtract(new BigDecimal("0.1")) $8 ==> 0.2
- qsort 6y agoYeah, and using float literals you get 0.20000...2: jshell> .3f-.1f $1 ==> 0.20000002 We got ourselves a hat trick :)