6 ms·
Switching to Python instead of Perl, use the 'statistics' module, https://docs.python.org/3/library/statistics.html https://docs.python.org/3/library/statistics
by eesmith 5y ago
Switching to Python instead of Perl, use the 'statistics' module, https://docs.python.org/3/library/statistics.html https://docs.python.org/3/library/statistics.html :
>>> data = (1_000_000_000.1, 1.1) * 50_000
>>> sum(data) / len(data)
500000000.60091573
>>> sum(sorted(data)) / len(data)
500000000.6004578
>>> sum(sorted(data, reverse=True)) / len(data)
500000000.6012391
>>> import statistics
>>> statistics.mean(data)
500000000.6
and
>>> X = [10_000_000_000 + k for k in (4, 7, 13, 16)]
>>> statistics.variance(X)
30
- pdkl95 5y agoRuby seems to handle this out of the box: >> RUBY_VERSION ⇒ "2.7.2" >> data = [1_000_000_000.1, 1.1] * 50_000 ; nil ⇒ nil >> data.sum ⇒ 50000000060000.0 >> data.sum / data.length ⇒ 500000000.6 >> data.sort.sum / data.length ⇒ 500000000.6 >> data.sort.reverse.sum / data.length ⇒ 500000000.6 I know that ruby will auto-promote an Integer into a Bignum. It seems like ruby will also auto-promote other types? I should research this...
- dmurray 5y agoFWIW, Ruby the language doesn't have Bignum any more, though, just Integer which handles arbitrarily large numbers. Under the hood, there are some optimizations for small integers. https://bugs.ruby-lang.org/issues/12005 https://bugs.ruby-lang.org/issues/12005
- chrisseaton 5y agoBignum is still there, working in almost exactly the same way, it's not exposed to the user anymore.
- aardvark179 5y agoNope. The sum method is implemented in C, and as soon as it sees a float value in the array it switches to Kahan-Babuska summation. We don’t do that in TruffleRuby and so don’t produce quite such an accurate result.
- ska 5y agoThis is a nice thing about Common Lisp, not just integer promotion to bignums, but ratios too (works in complex, also).
- trinovantes 5y agoI guess the question is does it still use floats internally or use something like BigNum and handle the precision issue (and thus more memory/cpu)
- _delirium 5y agoIt looks like the second. For many of its operations, the statistics module first converts floats to integer ratios (integers are bignums in Python), does intermediate operations using integer arithmetic, and converts back to floats at the end. Here's the internal _sum function for example, which you can see maps _exact_ratio over the data before summing: https://github.com/python/cpython/blob/3.9/Lib/statistics.py#L123 https://github.com/python/cpython/blob/3.9/Lib/statistics.py...
- eesmith 5y agoThere's also fmean() which is for floats: >>> data = (1_000_000_000.1, 1.1) * 50_000 >>> statistics.fmean(data) 500000000.6 Implementation at https://github.com/python/cpython/blob/6df926f1c46eb6db7b5dcd0227c6b532c78525c9/Modules/mathmodule.c#L1314 https://github.com/python/cpython/blob/6df926f1c46eb6db7b5dc... .
- eesmith 5y agoI just found that boltons supports a mean, in boltons.statsutils.mean . It appears to use the naive summation: >>> import boltons.statsutils >>> data = (1_000_000_000.1, 1.1) * 50_000 >>> boltons.statsutils.mean(data) 500000000.60091573 >>> sum(data) / len(data) 500000000.60091573