4 ms·

Given how slow Python is, isn't it embarrassing that 0.2 + 0.1 > 0.3 ? I have some test Rust code where I add up about a hundred million 32-bit floating point

Given how slow Python is, isn't it embarrassing that 0.2 + 0.1 > 0.3 ? I have some test Rust code where I add up about a hundred million 32-bit floating point numbers in the naive way, and it takes maybe a hundred milliseconds, and then I do the same but accumulating in a realistic::Real because hey how much slower is this type than a floating point number, well that's closer to twenty seconds. But if I ask Python to do this, Python takes about twenty seconds anyway, and yet it's using floating point arithmetic so it gets the sum wrong, whereas realistic::Real doesn't because it's storing the exact values.

shagie 1y ago
https://0.30000000000000004.com/#rust https://0.30000000000000004.com/#rust
- tialaramex 1y ago
  Er, yes, I'm aware why this happened, my point is that this happens in the hardware floating point, but Python is as slow as the non-accelerated big rationals in my realistic::Real (it's actually markedly slower than the more appropriate realistic::Rational but that's not what my existing benchmarks cared about)
  - rbanffy 1y ago
    > this happens in the hardware floating point Not really. It's a limitation of the IEEE floating point format used in most programming languages. Some numbers that look nice in base 10 don't have an exact representation in base 2.
    - shagie 1y ago
      Rational numbers where the denominator is a multiple of the prime factors of the base have an exact fractional representation in that base. 1/3 doesn't have an exact representation in base 10 or base 2. 1/5th does have an exact representation in base 10 (0.2), but doesn't in base 2. 1/4th has an exact representation in base 10 (0.25) and in base 2 (0.01)
rbanffy 1y ago
If you have a hundred million integers to add, please, by all means, use Rust, or C, and intrinsics for the features not yet in your favorite math libraries. You can call Rust from Python, for instance. Polars is excellent, BTW. This seamless integration between a nice language amenable to interactive experimentation and highly optimized code produced in many other languages I don't want to write code in is what makes Python an excellent choice in many business cases.
- Qem 1y ago
  At least with BigInts, CPython implementation is quite fast. See https://www.wilfred.me.uk/blog/2014/10/20/the-fastest-bigint-in-the-west/ https://www.wilfred.me.uk/blog/2014/10/20/the-fastest-bigint...
zahlman 1y ago
>But if I ask Python to [add up about a hundred million] [32-bit floating point numbers in the naive way], Python takes about twenty seconds anyway Really? $ python -m timeit -s 'import random; x = [random.random() for _ in range(100000000)]' 'sum(x)' 1 loop, best of 5: 806 msec per loop This is on 11-year-old hardware. Even the generation isn't that slow: $ time python -c 'import random; x = [random.random() for _ in range(100000000)]' real 0m10.942s user 0m9.590s sys 0m1.346s Of course, Python forces the use of doubles internally. Any optimizations inherent to 32-bit floats are simply not available.
eesmith 1y ago
You shouldn't add hundred million 32-bit floating point numbers in the naive way. You should use Kahan or Neumaier summation. In Python these are available as math.fsum() and (in recent Python releases) the built-in sum function. If you did total = 0.0 for value in data: total += value instead of total = sum(data) then yes, the answer will take longer and be less accurate. But the naive native Rust equivalent will be less accurate than Python's sum(data).
- tialaramex 1y ago
  > You shouldn't add hundred million 32-bit floating point numbers in the naive way. That's entirely correct, you shouldn't do this. And yet people do for one reason and another. I'm aware of Kahan summation (and Neumaier's improvement), but it wasn't the point of the benchmarks I happened to be performing when this topic arrived. You will not be surprised to learn there's a Kahan adaptor crate for Rust's iterator, so (with that crate) you can ask for the Kahan sum of some floating point iterator just the same way as you could ask for the naive sum. I suppose it's conceivable that one day Rust will choose to ship a specialisation in the stdlib which uses Kahan (as Python did in newer versions) but that seems unlikely because it is slower and you could explicitly ask for it already if you need it.
  - eesmith 1y ago
    Let me see if I have it right. You don't like Python's use of IEEE 754 float64 for its "float" type because it's already so slow that you think Python should use a data type which better fits the expectations of primary school math training. Then to demonstrate the timing issue you give an example where you ignore the simplest, fastest, and most accurate Python solution, using a built-in function which would likely be more accurate than what's available in stock Rust. If accuracy is important for the first, why is it not important for the second? Are you aware of the long history of compiled versions of Python (PyPy, numba, and more), plus variants like Cython, where the result has near Rust performance levels? Were the core float be a non-IEEE 754, those compiled versions would either be dog slow (to be compatible with Python's core float) or give results which are different than CPython's. Even if they did not exist, there would be a lot of questions about why a given program in C, Rust, Pascal, or any other float64-based system, gives different answers when translated to Python. FWIW, I, like others, could not reproduce your 20 seconds timing. Python is indeed slow for this sort of task, but even for explicit iteration my code is an order of magnitude faster than you reported.

shagie 1y ago

https://0.30000000000000004.com/#rust https://0.30000000000000004.com/#rust

tialaramex 1y ago
Er, yes, I'm aware why this happened, my point is that this happens in the hardware floating point, but Python is as slow as the non-accelerated big rationals in my realistic::Real (it's actually markedly slower than the more appropriate realistic::Rational but that's not what my existing benchmarks cared about)
- rbanffy 1y ago
  > this happens in the hardware floating point Not really. It's a limitation of the IEEE floating point format used in most programming languages. Some numbers that look nice in base 10 don't have an exact representation in base 2.
  - shagie 1y ago
    Rational numbers where the denominator is a multiple of the prime factors of the base have an exact fractional representation in that base. 1/3 doesn't have an exact representation in base 10 or base 2. 1/5th does have an exact representation in base 10 (0.2), but doesn't in base 2. 1/4th has an exact representation in base 10 (0.25) and in base 2 (0.01)

rbanffy 1y ago

If you have a hundred million integers to add, please, by all means, use Rust, or C, and intrinsics for the features not yet in your favorite math libraries. You can call Rust from Python, for instance. Polars is excellent, BTW. This seamless integration between a nice language amenable to interactive experimentation and highly optimized code produced in many other languages I don't want to write code in is what makes Python an excellent choice in many business cases.

Qem 1y ago
At least with BigInts, CPython implementation is quite fast. See https://www.wilfred.me.uk/blog/2014/10/20/the-fastest-bigint-in-the-west/ https://www.wilfred.me.uk/blog/2014/10/20/the-fastest-bigint...

zahlman 1y ago

>But if I ask Python to [add up about a hundred million] [32-bit floating point numbers in the naive way], Python takes about twenty seconds anyway Really? $ python -m timeit -s 'import random; x = [random.random() for _ in range(100000000)]' 'sum(x)' 1 loop, best of 5: 806 msec per loop This is on 11-year-old hardware. Even the generation isn't that slow: $ time python -c 'import random; x = [random.random() for _ in range(100000000)]' real 0m10.942s user 0m9.590s sys 0m1.346s Of course, Python forces the use of doubles internally. Any optimizations inherent to 32-bit floats are simply not available.

eesmith 1y ago

You shouldn't add hundred million 32-bit floating point numbers in the naive way. You should use Kahan or Neumaier summation. In Python these are available as math.fsum() and (in recent Python releases) the built-in sum function. If you did total = 0.0 for value in data: total += value instead of total = sum(data) then yes, the answer will take longer and be less accurate. But the naive native Rust equivalent will be less accurate than Python's sum(data).

tialaramex 1y ago
> You shouldn't add hundred million 32-bit floating point numbers in the naive way. That's entirely correct, you shouldn't do this. And yet people do for one reason and another. I'm aware of Kahan summation (and Neumaier's improvement), but it wasn't the point of the benchmarks I happened to be performing when this topic arrived. You will not be surprised to learn there's a Kahan adaptor crate for Rust's iterator, so (with that crate) you can ask for the Kahan sum of some floating point iterator just the same way as you could ask for the naive sum. I suppose it's conceivable that one day Rust will choose to ship a specialisation in the stdlib which uses Kahan (as Python did in newer versions) but that seems unlikely because it is slower and you could explicitly ask for it already if you need it.
- eesmith 1y ago
  Let me see if I have it right. You don't like Python's use of IEEE 754 float64 for its "float" type because it's already so slow that you think Python should use a data type which better fits the expectations of primary school math training. Then to demonstrate the timing issue you give an example where you ignore the simplest, fastest, and most accurate Python solution, using a built-in function which would likely be more accurate than what's available in stock Rust. If accuracy is important for the first, why is it not important for the second? Are you aware of the long history of compiled versions of Python (PyPy, numba, and more), plus variants like Cython, where the result has near Rust performance levels? Were the core float be a non-IEEE 754, those compiled versions would either be dog slow (to be compatible with Python's core float) or give results which are different than CPython's. Even if they did not exist, there would be a lot of questions about why a given program in C, Rust, Pascal, or any other float64-based system, gives different answers when translated to Python. FWIW, I, like others, could not reproduce your 20 seconds timing. Python is indeed slow for this sort of task, but even for explicit iteration my code is an order of magnitude faster than you reported.