2 ms·
something like, in mysql: create table t ( value double(10,2) not null ); put a bunch of values, something like 15k with maybe half of them being 0.37. Do the
by radu_floricica 4y ago
something like, in mysql:
create table t (
value double(10,2) not null
);
put a bunch of values, something like 15k with maybe half of them being 0.37.
Do the sum two ways:
- export them in excel and do a sum
- select sum(value) from t
and you get two different values, with a difference of something like 15.
Replace double(10,2) with decimal(10,2), and the sum(value) works correctly.
I still have no idea why this happens. I could expect a tiny difference, and code like your above will indeed give differences like this, well under 1.0, but my particular scenario had differences many orders of magnitude higher. Still can't explain.
- archi42 4y agosomething like 15k with maybe half of them being 0.37 Oh, that's something entirely different. If some of your other numbers are much larger than 0.37, you will run into a different kind of problem. You know about exponent and mantissa? Adding a value with a small exponent to a value with a big exponent will cause imprecisions. In extreme cases the result is just the bigger number.