3 ms·

Floating point arithmetic is neither commutative or associative so you shouldn’t.

lo0dot0 1y ago
While it technically correct to say this it also gets the wrong point across because it leaves out the fact that ordering changes create only a small difference. Other examples where arithmetic is not commutative, e.g. matrix multiplication , can create much larger differences.
- kstrauser 1y ago
  > ordering changes create only a small difference. That can’t be assumed. You can easily fall into a situation like: total = large_float_value for _ in range(1_000_000_000): total += .01 assert total == large_float_value Without knowing the specific situation, it’s impossible to say whether that’s a tolerably small difference.
eapriv 1y ago
Why is it not commutative?
- layer8 1y ago
  It actually is commutative according to IEEE-754, except that in the case of a NaN result you might get a different NaN representation.
  - adgjlsfhk1 1y ago
    having multiple NaNs and no spec for how they should behave feels like such an unforced error to me
    - layer8 1y ago
      For mathematical use, NaN payloads shouldn’t matter, and behave identically (aside from quiet vs. signaling NaNs). It also doesn’t matter for equality comparison, because NaNs always compare unequal.
      - adgjlsfhk1 1y ago
        from the user perspective it's not too bad, but from the compiler perspective it is. The result of this is that LLVM has decided that trying to figure out which nan you got (e.g. by casting to an Int and comparing) is UB, which means pretty much every floating point operation becomes non-deterministic. This also adds extra complexity to the CPU. you need special hardware for == rather than just using the perfectly good integer unit, and every fpu operation needs to devote a bunch of transistors to handling this nonsense that buys the user absolutely nothing. there are definitely things to criticize about the design of Posits, but the thing they 100% get right is having a single NaN and sane ordering semantics
layer8 1y ago
IEEE-754 addition and multiplication is commutative. It isn't distributive, though.
StefanKarpinski 1y ago
Floating-point arithmetic is non-associative, but it is commutative for the operations that are algebraically commutative: x + y == y + x and x*y == y*x. And x - y = -(y - x) so subtraction is properly anti-commutative. The only very marginal exception to this is that when both arguments are NaN, the return value will be NaN, but which NaN payload is returned can depend on argument order. But no one ever uses this because it's not specified, so it can't be used reliably for anything useful. The behavior I wish IEEE 754 had specified for this is to define a standard NaN value (or two), and when the return value of an op is NaN, and some of the arguments are non-standard NaNs, then one of those non-standard NaN values must be returned. This doesn't depend on argument order and allows NaN payloads to be reliably propagated, which would let you encode useful debugging information in NaN payloads and know that it will flow through the program.

While it technically correct to say this it also gets the wrong point across because it leaves out the fact that ordering changes create only a small difference. Other examples where arithmetic is not commutative, e.g. matrix multiplication , can create much larger differences.

kstrauser 1y ago
> ordering changes create only a small difference. That can’t be assumed. You can easily fall into a situation like: total = large_float_value for _ in range(1_000_000_000): total += .01 assert total == large_float_value Without knowing the specific situation, it’s impossible to say whether that’s a tolerably small difference.

eapriv 1y ago

Why is it not commutative?

layer8 1y ago
It actually is commutative according to IEEE-754, except that in the case of a NaN result you might get a different NaN representation.
- adgjlsfhk1 1y ago
  having multiple NaNs and no spec for how they should behave feels like such an unforced error to me
  - layer8 1y ago
    For mathematical use, NaN payloads shouldn’t matter, and behave identically (aside from quiet vs. signaling NaNs). It also doesn’t matter for equality comparison, because NaNs always compare unequal.
    - adgjlsfhk1 1y ago
      from the user perspective it's not too bad, but from the compiler perspective it is. The result of this is that LLVM has decided that trying to figure out which nan you got (e.g. by casting to an Int and comparing) is UB, which means pretty much every floating point operation becomes non-deterministic. This also adds extra complexity to the CPU. you need special hardware for == rather than just using the perfectly good integer unit, and every fpu operation needs to devote a bunch of transistors to handling this nonsense that buys the user absolutely nothing. there are definitely things to criticize about the design of Posits, but the thing they 100% get right is having a single NaN and sane ordering semantics

layer8 1y ago

IEEE-754 addition and multiplication is commutative. It isn't distributive, though.

StefanKarpinski 1y ago

Floating-point arithmetic is non-associative, but it is commutative for the operations that are algebraically commutative: x + y == y + x and x*y == y*x. And x - y = -(y - x) so subtraction is properly anti-commutative. The only very marginal exception to this is that when both arguments are NaN, the return value will be NaN, but which NaN payload is returned can depend on argument order. But no one ever uses this because it's not specified, so it can't be used reliably for anything useful. The behavior I wish IEEE 754 had specified for this is to define a standard NaN value (or two), and when the return value of an op is NaN, and some of the arguments are non-standard NaNs, then one of those non-standard NaN values must be returned. This doesn't depend on argument order and allows NaN payloads to be reliably propagated, which would let you encode useful debugging information in NaN payloads and know that it will flow through the program.