3 ms·

You shouldn't add hundred million 32-bit floating point numbers in the naive way. You should use Kahan or Neumaier summation. In Python these are available as m

You shouldn't add hundred million 32-bit floating point numbers in the naive way. You should use Kahan or Neumaier summation. In Python these are available as math.fsum() and (in recent Python releases) the built-in sum function. If you did total = 0.0 for value in data: total += value instead of total = sum(data) then yes, the answer will take longer and be less accurate. But the naive native Rust equivalent will be less accurate than Python's sum(data).

tialaramex 1y ago
> You shouldn't add hundred million 32-bit floating point numbers in the naive way. That's entirely correct, you shouldn't do this. And yet people do for one reason and another. I'm aware of Kahan summation (and Neumaier's improvement), but it wasn't the point of the benchmarks I happened to be performing when this topic arrived. You will not be surprised to learn there's a Kahan adaptor crate for Rust's iterator, so (with that crate) you can ask for the Kahan sum of some floating point iterator just the same way as you could ask for the naive sum. I suppose it's conceivable that one day Rust will choose to ship a specialisation in the stdlib which uses Kahan (as Python did in newer versions) but that seems unlikely because it is slower and you could explicitly ask for it already if you need it.
- eesmith 1y ago
  Let me see if I have it right. You don't like Python's use of IEEE 754 float64 for its "float" type because it's already so slow that you think Python should use a data type which better fits the expectations of primary school math training. Then to demonstrate the timing issue you give an example where you ignore the simplest, fastest, and most accurate Python solution, using a built-in function which would likely be more accurate than what's available in stock Rust. If accuracy is important for the first, why is it not important for the second? Are you aware of the long history of compiled versions of Python (PyPy, numba, and more), plus variants like Cython, where the result has near Rust performance levels? Were the core float be a non-IEEE 754, those compiled versions would either be dog slow (to be compatible with Python's core float) or give results which are different than CPython's. Even if they did not exist, there would be a lot of questions about why a given program in C, Rust, Pascal, or any other float64-based system, gives different answers when translated to Python. FWIW, I, like others, could not reproduce your 20 seconds timing. Python is indeed slow for this sort of task, but even for explicit iteration my code is an order of magnitude faster than you reported.
  - tialaramex 1y ago
    I think that a language which (as I understand it) already gives you a big number if you exceed the bounds of its provided integer types, might just as well provide rationals as the IEEE floating point types unless those IEEE types are very fast but to me it looks like on the whole they're not. I was not aware of Cython, which sounds like a terrible idea but each to his own nor Numba, though I have worked with PyPy in the past. I'm afraid that the idea that somehow every Python implementation would be compatible caused me to choke. Python doesn't really care about compatibility, behaviour of the sum function you brought up was changed twice since Python 3. This is an old machine, so I can well believe you can do the iteration faster. My initial interest happened because by total coincidence I was writing benchmarks for realistic which try out f32 and f64 vs realistic::Real for various arithmetic operations, and so I wondered well, isn't even Python much faster and (with my naive iteration) it was not. As you are hopefully aware, new programmers are equally likely to run into languages where the default is the 32-bit IEEE floating point and so 1.0 + 2.0 > 3.0 is false for them as they are to encounter a language like Python with 64-bit IEEE floats. I'd expect, as with Python's experience with their hash tables, the kind of people writing Python as their main or even only language would always be pleased to have simpler, less surprising behaviour, and the rationals are much simpler - they're just slower.
    - eesmith 1y ago
      The int/long unification occurred a long time ago. ("long" was Python's name for BigNum.) There is now no such thing as "exceed the bounds of its provided integer types". Guido van Rossum, who started and lead the Python project for many years, previously worked with ABC, which used rational as the default type. In practice this caused problems as it was all to easy to end up with "pathologically large numerators and denominators" (quoting https://docs.raku.org/type/FatRat https://docs.raku.org/type/FatRat). That experience guided him to reject rationals as the default integer type. Pathologically large numerators and denominators make rationals not "just slower" but "a lot slower". > somehow every Python implementation would be compatible It's more of a rough consensus thing than full compatibility. > Python doesn't really care about compatibility Correct, and it causes me pain every years. But do note that historic compatibility is different than cross-implementation compatibility, since there is a strong incentive for other implementations to do a good job of staying CPython compatible. FWIW, the laptop where I did my timings is 5 years old. The new programmers in my field generally have Python as their first language, and don't have experience with float32. I also know that float32 isn't enough to distinguish rationals I need to deal with, since in float32, 2094/4097 == 2117/4142 == 0.511106, while in float64 those ratios are not equal, as 0.5111056870881132 != 0.5111057460164172. (I internally use uint16_t for the ratios, but have to turn them into doubles for use by other tools.)

tialaramex 1y ago

> You shouldn't add hundred million 32-bit floating point numbers in the naive way. That's entirely correct, you shouldn't do this. And yet people do for one reason and another. I'm aware of Kahan summation (and Neumaier's improvement), but it wasn't the point of the benchmarks I happened to be performing when this topic arrived. You will not be surprised to learn there's a Kahan adaptor crate for Rust's iterator, so (with that crate) you can ask for the Kahan sum of some floating point iterator just the same way as you could ask for the naive sum. I suppose it's conceivable that one day Rust will choose to ship a specialisation in the stdlib which uses Kahan (as Python did in newer versions) but that seems unlikely because it is slower and you could explicitly ask for it already if you need it.

eesmith 1y ago
Let me see if I have it right. You don't like Python's use of IEEE 754 float64 for its "float" type because it's already so slow that you think Python should use a data type which better fits the expectations of primary school math training. Then to demonstrate the timing issue you give an example where you ignore the simplest, fastest, and most accurate Python solution, using a built-in function which would likely be more accurate than what's available in stock Rust. If accuracy is important for the first, why is it not important for the second? Are you aware of the long history of compiled versions of Python (PyPy, numba, and more), plus variants like Cython, where the result has near Rust performance levels? Were the core float be a non-IEEE 754, those compiled versions would either be dog slow (to be compatible with Python's core float) or give results which are different than CPython's. Even if they did not exist, there would be a lot of questions about why a given program in C, Rust, Pascal, or any other float64-based system, gives different answers when translated to Python. FWIW, I, like others, could not reproduce your 20 seconds timing. Python is indeed slow for this sort of task, but even for explicit iteration my code is an order of magnitude faster than you reported.
- tialaramex 1y ago
  I think that a language which (as I understand it) already gives you a big number if you exceed the bounds of its provided integer types, might just as well provide rationals as the IEEE floating point types unless those IEEE types are very fast but to me it looks like on the whole they're not. I was not aware of Cython, which sounds like a terrible idea but each to his own nor Numba, though I have worked with PyPy in the past. I'm afraid that the idea that somehow every Python implementation would be compatible caused me to choke. Python doesn't really care about compatibility, behaviour of the sum function you brought up was changed twice since Python 3. This is an old machine, so I can well believe you can do the iteration faster. My initial interest happened because by total coincidence I was writing benchmarks for realistic which try out f32 and f64 vs realistic::Real for various arithmetic operations, and so I wondered well, isn't even Python much faster and (with my naive iteration) it was not. As you are hopefully aware, new programmers are equally likely to run into languages where the default is the 32-bit IEEE floating point and so 1.0 + 2.0 > 3.0 is false for them as they are to encounter a language like Python with 64-bit IEEE floats. I'd expect, as with Python's experience with their hash tables, the kind of people writing Python as their main or even only language would always be pleased to have simpler, less surprising behaviour, and the rationals are much simpler - they're just slower.
  - eesmith 1y ago
    The int/long unification occurred a long time ago. ("long" was Python's name for BigNum.) There is now no such thing as "exceed the bounds of its provided integer types". Guido van Rossum, who started and lead the Python project for many years, previously worked with ABC, which used rational as the default type. In practice this caused problems as it was all to easy to end up with "pathologically large numerators and denominators" (quoting https://docs.raku.org/type/FatRat https://docs.raku.org/type/FatRat). That experience guided him to reject rationals as the default integer type. Pathologically large numerators and denominators make rationals not "just slower" but "a lot slower". > somehow every Python implementation would be compatible It's more of a rough consensus thing than full compatibility. > Python doesn't really care about compatibility Correct, and it causes me pain every years. But do note that historic compatibility is different than cross-implementation compatibility, since there is a strong incentive for other implementations to do a good job of staying CPython compatible. FWIW, the laptop where I did my timings is 5 years old. The new programmers in my field generally have Python as their first language, and don't have experience with float32. I also know that float32 isn't enough to distinguish rationals I need to deal with, since in float32, 2094/4097 == 2117/4142 == 0.511106, while in float64 those ratios are not equal, as 0.5111056870881132 != 0.5111057460164172. (I internally use uint16_t for the ratios, but have to turn them into doubles for use by other tools.)