4 ms·

Yes, "floating point accumulation doesn't commute" is a mantra everyone should have in their head, and when I first read this article, I was jumping at the bit

by bri3d 8mo ago

Yes, "floating point accumulation doesn't commute" is a mantra everyone should have in their head, and when I first read this article, I was jumping at the bit to dismiss it out of hand for that reason. But, what got me about this is that: * every other Apple device delivered the same results * Apple's own LLM silently failed on this device to me that behavior suggests an unexpected failure rather than a fundamental issue; it seems Bad (TM) that Apple would ship devices where their own LLM didn't work.

danpalmer 8mo ago
FYI, the saying is "champing at the bit", it comes from horses being restrained.
- mylifeandtimes 8mo ago
  hey, I appreciate your love of language and sharing with us. I'm wondering if we couldn't re-think "bit" to the computer science usage instead of the thing that goes in the horse's mouth, and what it would mean for an AI agent to "champ at the bit"? What new sayings will we want?
  - nilamo 8mo ago
    Byting at the bit?
- odo1242 8mo ago
  chomping at the bit
  - danpalmer 8mo ago
    Actually it was originally "champing" – to grind or gnash teeth. The "chomping" (to bite) alternative cropped up more recently as people misheard and misunderstood, but it's generally accepted as an alternative now.
    - kortilla 8mo ago
      It’s actually accepted as the primary now and telling people about “champing” is just seen as archaic.
      - danpalmer 8mo ago
        Do you have a source on this, or a definition for what it means to be "primary" here? All I can find is sources confirming that "champing" is the original and more technically correct, but that "chomping" is an accepted variant.
    - odo1242 8mo ago
      I see
- jasinjames 8mo ago
  Huh. I never knew "champing" was the proper spelling [0] [0] https://www.npr.org/sections/memmos/2016/06/09/605796769/chew-on-this-is-it-chomping-or-champing https://www.npr.org/sections/memmos/2016/06/09/605796769/che...
sva_ 8mo ago
> floating point accumulation doesn't commute It is commutative (except for NaN). It isn't associative though.
- ekelsen 8mo ago
  I think it commutes even when one or both inputs are NaN? The output is always NaN.
  - addaon 8mo ago
    NaNs are distinguishable. /Which/ NaN you get doesn't commute.
    - ekelsen 8mo ago
      I guess at the bit level, but not at the level of computation? Anything that relies on bit patterns of nans behaving in a certain way (like how they propagate) is in dangerous territory.
      - addaon 8mo ago
        > Anything that relies on bit patterns of nans behaving in a certain way (like how they propagate) is in dangerous territory. Why? This is well specified by IEEE 754. Many runtimes (e.g. for Javascript) use NaN boxing. Treating floats as a semi-arbitrary selection of rational numbers plus a handful of special values is /more/ correct than treating them as real numbers, but treating them as actually specified does give more flexibility and power.
        ekelsen 8mo ago
        Can you show me where in the ieee spec this is guaranteed? My understanding is the exact opposite - that it allows implementations to return any NaN value at all. It need not be any that were inputs. It may be that JavaScript relies on it and that has become more binding than the actual spec, but I don't think the spec actually guarantees this. Edit: actually it turns out nan-boxing does not involve arithmetic, which is why it works. I think my original point stands, if you are doing something that relies on how bit values of NaNs are propagated during arithmetic, you are on shaky ground.
BeetleB 8mo ago
As a sister comment said, floating point computations are commutative, but not associative. a * b = b * a for all "normal" floating point numbers.
DavidVoid 8mo ago
I would go even further and state that "you should never assume that floating point functions will evaluate the same on two different computers, or even on two different versions of the same application", as the results of floating point evaluations can differ depending on platform, compiler optimizations, compilation-flags, run-time FPU environment (rounding mode, &c.), and even memory alignment of run-time data. There's a C++26 paper about compile time math optimizations with a good overview and discussion about some of these issues [P1383]. The paper explicitly states: 1. It is acceptable for evaluation of mathematical functions to differ between translation time and runtime. 2. It is acceptable for constant evaluation of mathematical functions to differ between platforms. So C++ has very much accepted the fact that floating point functions should not be presumed to give identical results in all circumstances. Now, it is of course possible to ensure that floating point-related functions give identical results on all your target machines, but it's usually not worth the hassle. [P1383]: https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2023/p1383r2.pdf https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2023/p13...
- physicsguy 8mo ago
  Even the exact same source code compiled with different compilers, or the same compiler with different compiler options. Intel Compiler for e.g. uses less than IEEE764 precision for floating point ops by default, for example.

by bri3d 8mo ago

danpalmer 8mo ago

FYI, the saying is "champing at the bit", it comes from horses being restrained.

mylifeandtimes 8mo ago
hey, I appreciate your love of language and sharing with us. I'm wondering if we couldn't re-think "bit" to the computer science usage instead of the thing that goes in the horse's mouth, and what it would mean for an AI agent to "champ at the bit"? What new sayings will we want?
- nilamo 8mo ago
  Byting at the bit?
odo1242 8mo ago
chomping at the bit
- danpalmer 8mo ago
  Actually it was originally "champing" – to grind or gnash teeth. The "chomping" (to bite) alternative cropped up more recently as people misheard and misunderstood, but it's generally accepted as an alternative now.
  - kortilla 8mo ago
    It’s actually accepted as the primary now and telling people about “champing” is just seen as archaic.
    - danpalmer 8mo ago
      Do you have a source on this, or a definition for what it means to be "primary" here? All I can find is sources confirming that "champing" is the original and more technically correct, but that "chomping" is an accepted variant.
  - odo1242 8mo ago
    I see
jasinjames 8mo ago
Huh. I never knew "champing" was the proper spelling [0] [0] https://www.npr.org/sections/memmos/2016/06/09/605796769/chew-on-this-is-it-chomping-or-champing https://www.npr.org/sections/memmos/2016/06/09/605796769/che...

sva_ 8mo ago

> floating point accumulation doesn't commute It is commutative (except for NaN). It isn't associative though.

ekelsen 8mo ago
I think it commutes even when one or both inputs are NaN? The output is always NaN.
- addaon 8mo ago
  NaNs are distinguishable. /Which/ NaN you get doesn't commute.
  - ekelsen 8mo ago
    I guess at the bit level, but not at the level of computation? Anything that relies on bit patterns of nans behaving in a certain way (like how they propagate) is in dangerous territory.
    - addaon 8mo ago
      > Anything that relies on bit patterns of nans behaving in a certain way (like how they propagate) is in dangerous territory. Why? This is well specified by IEEE 754. Many runtimes (e.g. for Javascript) use NaN boxing. Treating floats as a semi-arbitrary selection of rational numbers plus a handful of special values is /more/ correct than treating them as real numbers, but treating them as actually specified does give more flexibility and power.
      - ekelsen 8mo ago
        Can you show me where in the ieee spec this is guaranteed? My understanding is the exact opposite - that it allows implementations to return any NaN value at all. It need not be any that were inputs. It may be that JavaScript relies on it and that has become more binding than the actual spec, but I don't think the spec actually guarantees this. Edit: actually it turns out nan-boxing does not involve arithmetic, which is why it works. I think my original point stands, if you are doing something that relies on how bit values of NaNs are propagated during arithmetic, you are on shaky ground.

BeetleB 8mo ago

As a sister comment said, floating point computations are commutative, but not associative. a * b = b * a for all "normal" floating point numbers.

DavidVoid 8mo ago

I would go even further and state that "you should never assume that floating point functions will evaluate the same on two different computers, or even on two different versions of the same application", as the results of floating point evaluations can differ depending on platform, compiler optimizations, compilation-flags, run-time FPU environment (rounding mode, &c.), and even memory alignment of run-time data. There's a C++26 paper about compile time math optimizations with a good overview and discussion about some of these issues [P1383]. The paper explicitly states: 1. It is acceptable for evaluation of mathematical functions to differ between translation time and runtime. 2. It is acceptable for constant evaluation of mathematical functions to differ between platforms. So C++ has very much accepted the fact that floating point functions should not be presumed to give identical results in all circumstances. Now, it is of course possible to ensure that floating point-related functions give identical results on all your target machines, but it's usually not worth the hassle. [P1383]: https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2023/p1383r2.pdf https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2023/p13...

physicsguy 8mo ago
Even the exact same source code compiled with different compilers, or the same compiler with different compiler options. Intel Compiler for e.g. uses less than IEEE764 precision for floating point ops by default, for example.