26 ms·

I helped design an API for "algebraic operations" in Rust: <https://github.com/rust-lang/rust/issues/136469 https://github.com/rust-lang/rust/issues/136469>, wh

by orlp 1y ago

I helped design an API for "algebraic operations" in Rust: <https://github.com/rust-lang/rust/issues/136469 https://github.com/rust-lang/rust/issues/136469>, which are coming along nicely. These operations are 1. Localized, not a function-wide or program-wide flag. 2. Completely safe, -ffast-math includes assumptions such that there are no NaNs, and violating that is undefined behavior. So what do these algebraic operations do? Well, one by itself doesn't do much of anything compared to a regular operation. But a sequence of them is allowed to be transformed using optimizations which are algebraically justified, as-if all operations are done using real arithmetic.

eqvinox 1y ago
Are these calls going to clear the FTZ and DAZ flags in the MXCSR on x86? And FZ & FIZ in the FPCR on ARM?
- orlp 1y ago
  I don't believe so, no. Currently these operations only set the LLVM flags to allow reassociation, contraction, division replaced by reciprocal multiplication, and the assumption of no signed zeroes. This can be expanded in the future as LLVM offers more flags that fall within the scope of algebraically motivated optimizations.
  - eqvinox 1y ago
    Ah sorry I misunderstood and thought this API was for the other way around, i.e. forbidding "unsafe" operations. (I guess the question reverses to setting those flags) ('Naming: "algebraic" is not very descriptive of what this does since the operations themselves are algebraic.' :D)
    - nextaccountic 1y ago
      > ('Naming: "algebraic" is not very descriptive of what this does since the operations themselves are algebraic.' :D) Okay, the floating point operations are literally algebraic (they form an algebra) but they don't follow some common algebraic properties like associativity. The linked tracking issue itself acknowledges that: > Naming: "algebraic" is not very descriptive of what this does since the operations themselves are algebraic. Also this comment https://github.com/rust-lang/rust/issues/136469#issuecomment-2727298355 https://github.com/rust-lang/rust/issues/136469#issuecomment... > > On that note I added an unresolved question for naming since algebraic isn't the most clear indicator of what is going on. > > I think it is fairly clear. The operations allow algebraically justified optimizations, as-if the arithmetic was real arithmetic. > > I don't think you're going to find a clearer name, but feel free to provide suggestions. One alternative one might consider is real_add, real_sub, etc. Then retorted here https://github.com/rust-lang/rust/issues/136469#issuecomment-2727298355 https://github.com/rust-lang/rust/issues/136469#issuecomment... > These names suggest that the operations are more accurate than normal, where really they are less accurate. One might misinterpret that these are infinite-precision operations (perhaps with rounding after a whole sequence of operations). > > The actual meaning isn't that these are real number operations, it's quite the opposite: they have best-effort precision with no strict guarantees. > > I find "algebraic" confusing for the same reason. > > How about approximate_add, approximate_sub? And the next comment > Saying "approximate" feels imperfect, as while these operations don't promise to produce the exact IEEE result on a per-operation basis, the overall result might well be more accurate algebraically. E.g.: > > (...) So there's a discussion going on about the naming
      - eqvinox 1y ago
        It doesn't feel appropriate to comment there for me not knowing any Rust really, but "lax_" (or "relax_") would have the extra benefit of being very short. (Is this going to overload operators or are people going to have to type this… a lot… ?)
        Sharlin 1y ago
        Rust has some precedence for adding convenience newtypes with overloaded operators (eg. `Wrapping<I>´ for `I.wrapping_add(I)` etc). Such a wrapper isn't currently proposed AFAIK but there's no reason one couldn't be added in the future I believe.
        eqvinox 1y ago
        Right, as long as the LLVM intrinsics are exposed you could just put that in a crate somewhere.
        Measter 1y ago
        For giggles, here's one I whipped up, along with an example use: https://godbolt.org/z/Eezj35dzc https://godbolt.org/z/Eezj35dzc
        Sharlin 1y ago
        Wow, that's some hardcore unrolling.
        eqvinox 1y ago
        Having done unspeakable things with the C preprocessor myself, this Rust macro soup truly warms my heart <3
        Measter 1y ago
        Oh this is simple, this is just a bit of basic expansion to generate boilerplate. The real experts can do some gnarly and amazing stuff with macros.
evrimoztamur 1y ago
Does that mean that a physics engine written with these operations will always compile to yield the same deterministic outcomes across different platforms (assuming they correctly implement (or able to do so) algebraic operations)?
- orlp 1y ago
  No, there is no guarantee which (if any) optimizations are applied, only that they may be applied. For example a fused multiply-add instruction may be emitted for a*b + c on platforms which support it, which is not cross-platform.
- SkiFire13 1y ago
  No, the result may depend on how the compiler reorders them, which could be different on different platforms.
- Sharlin 1y ago
  It's more like the opposite. These tell the compiler to assume for optimization purposes that floats are associative and so on (ie. algebraic), even when in reality they aren't. So the results may vary depending on what transformations the compiler performs – in particular, they may vary between optimized and non-optimized builds, which normally isn't allowed.
  - vanderZwan 1y ago
    > These tell the compiler to assume for optimization purposes that floats are associative and so on (ie. algebraic), even when in reality they aren't. I wonder if it is possible to add an additional constraint that guarantees the transformation has equal or fewer numerical rounding errors. E.g. for floating point doubles (0.2 + 0.1) - 0.1 results in 0.20000000000000004, so I would expect that transforming some (A + B) - B to just A would always reduce numerical error. OTOH, it's floating point maths, there's probably some kind of weird gotcha here as well.
    - legobmw99 1y ago
glkindlmann 1y ago
That sounds neat. What would be really neat is if the language helped to expose the consequences of the ensuing rounding error by automating things that are otherwise clumsy for programmers to do manually, like running twice with opposite rounding directions, or running many many times with internally randomized directions (two of the options in Sec 4 of *). That is, it would be cool if Rust enabled people learn about the subtleties of floating point, instead of hiding them away. * https://people.eecs.berkeley.edu/~wkahan/Mindless.pdf https://people.eecs.berkeley.edu/~wkahan/Mindless.pdf
pclmulqdq 1y ago
-ffast-math is actually something like 15 separate flags, and you can use them individually if you want. 3 of them are "no NaNs," "no infinities," and "no subnormals." Several of the other flags allow you to treat math as associative or distributive if you want that. The library has some merit, but the goal you've stated here is given to you with 5 compiler flags. The benefit of the library is choosing when these apply.
- foota 1y ago
  There's probably a benefit to being able to choose which you want in different places though, right?

by orlp 1y ago

eqvinox 1y ago

Are these calls going to clear the FTZ and DAZ flags in the MXCSR on x86? And FZ & FIZ in the FPCR on ARM?

orlp 1y ago
I don't believe so, no. Currently these operations only set the LLVM flags to allow reassociation, contraction, division replaced by reciprocal multiplication, and the assumption of no signed zeroes. This can be expanded in the future as LLVM offers more flags that fall within the scope of algebraically motivated optimizations.
- eqvinox 1y ago
  Ah sorry I misunderstood and thought this API was for the other way around, i.e. forbidding "unsafe" operations. (I guess the question reverses to setting those flags) ('Naming: "algebraic" is not very descriptive of what this does since the operations themselves are algebraic.' :D)
  - nextaccountic 1y ago
    > ('Naming: "algebraic" is not very descriptive of what this does since the operations themselves are algebraic.' :D) Okay, the floating point operations are literally algebraic (they form an algebra) but they don't follow some common algebraic properties like associativity. The linked tracking issue itself acknowledges that: > Naming: "algebraic" is not very descriptive of what this does since the operations themselves are algebraic. Also this comment https://github.com/rust-lang/rust/issues/136469#issuecomment-2727298355 https://github.com/rust-lang/rust/issues/136469#issuecomment... > > On that note I added an unresolved question for naming since algebraic isn't the most clear indicator of what is going on. > > I think it is fairly clear. The operations allow algebraically justified optimizations, as-if the arithmetic was real arithmetic. > > I don't think you're going to find a clearer name, but feel free to provide suggestions. One alternative one might consider is real_add, real_sub, etc. Then retorted here https://github.com/rust-lang/rust/issues/136469#issuecomment-2727298355 https://github.com/rust-lang/rust/issues/136469#issuecomment... > These names suggest that the operations are more accurate than normal, where really they are less accurate. One might misinterpret that these are infinite-precision operations (perhaps with rounding after a whole sequence of operations). > > The actual meaning isn't that these are real number operations, it's quite the opposite: they have best-effort precision with no strict guarantees. > > I find "algebraic" confusing for the same reason. > > How about approximate_add, approximate_sub? And the next comment > Saying "approximate" feels imperfect, as while these operations don't promise to produce the exact IEEE result on a per-operation basis, the overall result might well be more accurate algebraically. E.g.: > > (...) So there's a discussion going on about the naming
    - eqvinox 1y ago
      It doesn't feel appropriate to comment there for me not knowing any Rust really, but "lax_" (or "relax_") would have the extra benefit of being very short. (Is this going to overload operators or are people going to have to type this… a lot… ?)
      - Sharlin 1y ago
        Rust has some precedence for adding convenience newtypes with overloaded operators (eg. `Wrapping<I>´ for `I.wrapping_add(I)` etc). Such a wrapper isn't currently proposed AFAIK but there's no reason one couldn't be added in the future I believe.
        eqvinox 1y ago
        Right, as long as the LLVM intrinsics are exposed you could just put that in a crate somewhere.
        Measter 1y ago
        For giggles, here's one I whipped up, along with an example use: https://godbolt.org/z/Eezj35dzc https://godbolt.org/z/Eezj35dzc
        Sharlin 1y ago
        Wow, that's some hardcore unrolling.
        eqvinox 1y ago
        Having done unspeakable things with the C preprocessor myself, this Rust macro soup truly warms my heart <3
        Measter 1y ago
        Oh this is simple, this is just a bit of basic expansion to generate boilerplate. The real experts can do some gnarly and amazing stuff with macros.

evrimoztamur 1y ago

Does that mean that a physics engine written with these operations will always compile to yield the same deterministic outcomes across different platforms (assuming they correctly implement (or able to do so) algebraic operations)?

orlp 1y ago
No, there is no guarantee which (if any) optimizations are applied, only that they may be applied. For example a fused multiply-add instruction may be emitted for a*b + c on platforms which support it, which is not cross-platform.
SkiFire13 1y ago
No, the result may depend on how the compiler reorders them, which could be different on different platforms.
Sharlin 1y ago
It's more like the opposite. These tell the compiler to assume for optimization purposes that floats are associative and so on (ie. algebraic), even when in reality they aren't. So the results may vary depending on what transformations the compiler performs – in particular, they may vary between optimized and non-optimized builds, which normally isn't allowed.
- vanderZwan 1y ago
  > These tell the compiler to assume for optimization purposes that floats are associative and so on (ie. algebraic), even when in reality they aren't. I wonder if it is possible to add an additional constraint that guarantees the transformation has equal or fewer numerical rounding errors. E.g. for floating point doubles (0.2 + 0.1) - 0.1 results in 0.20000000000000004, so I would expect that transforming some (A + B) - B to just A would always reduce numerical error. OTOH, it's floating point maths, there's probably some kind of weird gotcha here as well.
  - legobmw99 1y ago

glkindlmann 1y ago

That sounds neat. What would be really neat is if the language helped to expose the consequences of the ensuing rounding error by automating things that are otherwise clumsy for programmers to do manually, like running twice with opposite rounding directions, or running many many times with internally randomized directions (two of the options in Sec 4 of *). That is, it would be cool if Rust enabled people learn about the subtleties of floating point, instead of hiding them away. * https://people.eecs.berkeley.edu/~wkahan/Mindless.pdf https://people.eecs.berkeley.edu/~wkahan/Mindless.pdf

pclmulqdq 1y ago

-ffast-math is actually something like 15 separate flags, and you can use them individually if you want. 3 of them are "no NaNs," "no infinities," and "no subnormals." Several of the other flags allow you to treat math as associative or distributive if you want that. The library has some merit, but the goal you've stated here is given to you with 5 compiler flags. The benefit of the library is choosing when these apply.

foota 1y ago
There's probably a benefit to being able to choose which you want in different places though, right?