4 ms·

What's the real-world consequences of having the floating point behavior changed? The article mentions some types of iterative algorithms but it's not clear how

What's the real-world consequences of having the floating point behavior changed? The article mentions some types of iterative algorithms but it's not clear how often those would be used. Would be interested to know what actual issues arose in any downstream projects.

adrian_b 2y ago
Computation errors that will be discovered only with great difficulties, when someone is careful enough to use some additional verification methods for the numeric results. The magnitude of the error caused by an underflow is completely unpredictable, which is why in any serious program it is unacceptable to ignore the underflows. Even when the errors happen to be discovered soon enough, people will be puzzled about their origin and they may waste a lot of time analyzing their algorithms, because there are a lot of things that can cause numeric errors. Because at their origin the underflows will neither signal any exception nor generate any unusual value, the errors will be normally caught much later, perhaps after thousands of other operations, when their cause will not be obvious. This fight between the people who want to get only correct results from computers and the people who do not care whether the results are erroneous as long as the results are obtained after a delay shorter by a few percents than for obtaining correct results has continued for decades, almost since electronic computers have been invented. Errors are acceptable in games or for some other graphics or audio applications that generate ephemeral images or sounds, but they are not acceptable for any engineering purposes. The option "-Ofast" is like smoking. There is no doubt that it is a bad habit, but whoever wants to smoke should be free to do it. On the other hand, exactly like a smoker should not be permitted to smoke in a closed room with non-smokers, the option "-Ofast" and the related options should not be permitted to alter the behavior of any other programs that are linked with an object file compiled with it. The behavior of "-Ofast" where it affects globally the content of MXCSR is unacceptable. The right behavior would have been for any function compiled with "-Ofast" to save MXCSR, put in it any desired value for the duration of the function execution and restore it at function exit. Moreover, when invoking any other function it should restore the original MXCSR before the call and put again in it the desired local value after the function returns back.
- bjourne 2y ago
  It's not so easy because -ffast-math can actually be more correct than running without.
  - adrian_b 2y ago
    In most cases it is unpredictable when such an improvement happens and what is unpredictable about correctness is useless. It is much more frequent for -ffast-math to generate bigger errors than to benefit from an accidental cancellation of the errors. The only case when the result of underflows is predictable is when the values whose computation may result in an underflow are immediately added to other values that are known to be big enough, and they are not used for anything else. It is not frequent to have so much information about which expressions will underflow and which will not, and about the range of possible values of the values that would be added. When there really exists so much information about what must be computed, then there are chances that the computation can be rearranged in a way that would avoid the underflows, in which case there would be no need to change MXCSR to ignore the underflows, because they would not happen.
    - bjourne 2y ago
      > It is much more frequent for -ffast-math to generate bigger errors than to benefit from an accidental cancellation of the errors. In my experience underflows are very uncommon and indicate that you're doing something wrong, whereas fp contractions are extremely common. So I disagree with you.
      - mbauman 2y ago
        fp contract isn't a uniform "decrease in error" either, though. As a simple example, it introduces error in the straightforward: a*b - a*b
      - adrian_b 2y ago
        I agree that using fused multiply-add in almost all cases improves the accuracy of the results. Nevertheless, I consider that a compiler option that gives you good FMA, but only together with bad behavior at underflows, is something exceedingly stupid. FMA is a standard operation, while ignoring underflows is rightly prohibited by the floating-point standard. There should never have been a compiler option that mixes standard operations with non-standard operations. In C or C++, you can use the "fma" function to get FMA where desired. This is more awkward, but it is safe. Any better solution requires a change in the programming language. For instance, it could be required that adjacent * and + must always be contracted into an FMA, unless separated by parentheses. This would allow the programmer to select between FMA and FMUL+FADD, as desired. FMA has appeared in 1990 (in IBM POWER) and it has been the greatest advance in floating-point computation after the introduction of Intel 8087 in 1980 (which has resulted in the IEEE standard a few years later). It is weird that 34 years later, most, if not all, programming languages do not handle well the generation of FMA operations.
mbauman 2y ago
In the case of flushing subnormals to zero, it's easy to end up with divides by zero when it wouldn't otherwise. `0/0.0` is `NaN` but `0/subnormal` is `Inf`. In other cases, `-ffast-math` just introduces arbitrary and strange behaviors. Sometimes you end up with higher precision than you expected. Other times you end up with less. Other times it'll helpfully just re-arrange things such that it's a zero. For example, the classical Kahan summation does the following: t = sum + y c = (t - sum) - y https://en.wikipedia.org/wiki/Kahan_summation_algorithm https://en.wikipedia.org/wiki/Kahan_summation_algorithm A -ffast-math compiler will see that — algebraically — you can just substitute `sum + y` into the equation for `c` and get 0. It's `sum + y - sum - y`. And that's true for real maths. But it's not true for floating point numbers. It explicitly destroys any attempt at _working with_ floating point numbers.
llsf 2y ago
No AI expert here, but my first thought was that it could be an attack vector to screw lots of AI projects.
coldpie 2y ago
Some arguably badly written programs depend on different execution contexts (across machines, processes, whatever) having consistent FP behavior for the same inputs. Having the FP behavior change out of your control can result in a desync across those contexts. Depending on the exact error behavior, these can be real hard to track down.
Willox 2y ago
I once accidentally inherited the fast math flag from a parent project in to builds of LuaJIT. The result was that LuaJIT's tagged unions, which are used pretty much everywhere and rely on normal fp functionality, broke drastically. Type checking for all values in the VM broke.
JonChesterfield 2y ago
Well, the program does different things. Some people might like that. Others might not. It's just one more factor in the generally observed quality of software.

adrian_b 2y ago

Computation errors that will be discovered only with great difficulties, when someone is careful enough to use some additional verification methods for the numeric results. The magnitude of the error caused by an underflow is completely unpredictable, which is why in any serious program it is unacceptable to ignore the underflows. Even when the errors happen to be discovered soon enough, people will be puzzled about their origin and they may waste a lot of time analyzing their algorithms, because there are a lot of things that can cause numeric errors. Because at their origin the underflows will neither signal any exception nor generate any unusual value, the errors will be normally caught much later, perhaps after thousands of other operations, when their cause will not be obvious. This fight between the people who want to get only correct results from computers and the people who do not care whether the results are erroneous as long as the results are obtained after a delay shorter by a few percents than for obtaining correct results has continued for decades, almost since electronic computers have been invented. Errors are acceptable in games or for some other graphics or audio applications that generate ephemeral images or sounds, but they are not acceptable for any engineering purposes. The option "-Ofast" is like smoking. There is no doubt that it is a bad habit, but whoever wants to smoke should be free to do it. On the other hand, exactly like a smoker should not be permitted to smoke in a closed room with non-smokers, the option "-Ofast" and the related options should not be permitted to alter the behavior of any other programs that are linked with an object file compiled with it. The behavior of "-Ofast" where it affects globally the content of MXCSR is unacceptable. The right behavior would have been for any function compiled with "-Ofast" to save MXCSR, put in it any desired value for the duration of the function execution and restore it at function exit. Moreover, when invoking any other function it should restore the original MXCSR before the call and put again in it the desired local value after the function returns back.

bjourne 2y ago
It's not so easy because -ffast-math can actually be more correct than running without.
- adrian_b 2y ago
  In most cases it is unpredictable when such an improvement happens and what is unpredictable about correctness is useless. It is much more frequent for -ffast-math to generate bigger errors than to benefit from an accidental cancellation of the errors. The only case when the result of underflows is predictable is when the values whose computation may result in an underflow are immediately added to other values that are known to be big enough, and they are not used for anything else. It is not frequent to have so much information about which expressions will underflow and which will not, and about the range of possible values of the values that would be added. When there really exists so much information about what must be computed, then there are chances that the computation can be rearranged in a way that would avoid the underflows, in which case there would be no need to change MXCSR to ignore the underflows, because they would not happen.
  - bjourne 2y ago
    > It is much more frequent for -ffast-math to generate bigger errors than to benefit from an accidental cancellation of the errors. In my experience underflows are very uncommon and indicate that you're doing something wrong, whereas fp contractions are extremely common. So I disagree with you.
    - mbauman 2y ago
      fp contract isn't a uniform "decrease in error" either, though. As a simple example, it introduces error in the straightforward: a*b - a*b
    - adrian_b 2y ago
      I agree that using fused multiply-add in almost all cases improves the accuracy of the results. Nevertheless, I consider that a compiler option that gives you good FMA, but only together with bad behavior at underflows, is something exceedingly stupid. FMA is a standard operation, while ignoring underflows is rightly prohibited by the floating-point standard. There should never have been a compiler option that mixes standard operations with non-standard operations. In C or C++, you can use the "fma" function to get FMA where desired. This is more awkward, but it is safe. Any better solution requires a change in the programming language. For instance, it could be required that adjacent * and + must always be contracted into an FMA, unless separated by parentheses. This would allow the programmer to select between FMA and FMUL+FADD, as desired. FMA has appeared in 1990 (in IBM POWER) and it has been the greatest advance in floating-point computation after the introduction of Intel 8087 in 1980 (which has resulted in the IEEE standard a few years later). It is weird that 34 years later, most, if not all, programming languages do not handle well the generation of FMA operations.

mbauman 2y ago

In the case of flushing subnormals to zero, it's easy to end up with divides by zero when it wouldn't otherwise. `0/0.0` is `NaN` but `0/subnormal` is `Inf`. In other cases, `-ffast-math` just introduces arbitrary and strange behaviors. Sometimes you end up with higher precision than you expected. Other times you end up with less. Other times it'll helpfully just re-arrange things such that it's a zero. For example, the classical Kahan summation does the following: t = sum + y c = (t - sum) - y https://en.wikipedia.org/wiki/Kahan_summation_algorithm https://en.wikipedia.org/wiki/Kahan_summation_algorithm A -ffast-math compiler will see that — algebraically — you can just substitute `sum + y` into the equation for `c` and get 0. It's `sum + y - sum - y`. And that's true for real maths. But it's not true for floating point numbers. It explicitly destroys any attempt at _working with_ floating point numbers.

llsf 2y ago

No AI expert here, but my first thought was that it could be an attack vector to screw lots of AI projects.

coldpie 2y ago

Some arguably badly written programs depend on different execution contexts (across machines, processes, whatever) having consistent FP behavior for the same inputs. Having the FP behavior change out of your control can result in a desync across those contexts. Depending on the exact error behavior, these can be real hard to track down.

Willox 2y ago

I once accidentally inherited the fast math flag from a parent project in to builds of LuaJIT. The result was that LuaJIT's tagged unions, which are used pretty much everywhere and rely on normal fp functionality, broke drastically. Type checking for all values in the VM broke.

JonChesterfield 2y ago

Well, the program does different things. Some people might like that. Others might not. It's just one more factor in the generally observed quality of software.