4 ms·
I'm curious about this. Could you provide an example? Here's what I'm seeing: --- JavaScript (function() { let inc = 0.37; let times = 5000;
by zeven7 4y ago
I'm curious about this. Could you provide an example?
Here's what I'm seeing:
---
JavaScript
(function() {
let inc = 0.37;
let times = 5000;
let total = 0;
for (let i = 0; i < 5000; i++) {
total += inc;
}
console.log('expected:', inc \* times);
console.log('actual:', total);
})();
expected: 1850
actual: 1849.9999999997679
---
Java
public class MyClass {
public static void main(String args[]) {
double inc = 0.37;
double times = 5000;
double total = 0;
for (double i = 0; i < 5000; i++) {
total += inc;
}
System.out.println("expected: " + (inc \* times));
System.out.println("actual: " + total);
}
}
expected: 1850.0
actual: 1849.9999999997679
---
update: Changing `double` to `float` in Java yields:
expected: 1850.0
actual: 1849.9778
and maybe that lines up with what you meant by "the differences should be a LOT farther than 3 digits", though it's hard to tell what you mean by "differences on the order of 10-20".
- HNDen21 4y agosame in SQL Server declare @i float =0.37,@times int = 0,@total float = 0 while @times < 5000 begin set @total+= @i set @times +=1 end select @total 1849.99999999977 or select sum(t) from( select top 5000 convert(float, 0.37) t from sysobjects a cross join sysobjects b) z 1849.99999999977
- radu_floricica 4y agosomething like, in mysql: create table t ( value double(10,2) not null ); put a bunch of values, something like 15k with maybe half of them being 0.37. Do the sum two ways: - export them in excel and do a sum - select sum(value) from t and you get two different values, with a difference of something like 15. Replace double(10,2) with decimal(10,2), and the sum(value) works correctly. I still have no idea why this happens. I could expect a tiny difference, and code like your above will indeed give differences like this, well under 1.0, but my particular scenario had differences many orders of magnitude higher. Still can't explain.
- archi42 4y agosomething like 15k with maybe half of them being 0.37 Oh, that's something entirely different. If some of your other numbers are much larger than 0.37, you will run into a different kind of problem. You know about exponent and mantissa? Adding a value with a small exponent to a value with a big exponent will cause imprecisions. In extreme cases the result is just the bigger number.