4 ms·

It's good to know there's a performance-accuracy tradeoff. Is the error the same in non-Apple targets? Does replacing FMADD with FNMADD (negative mult) or FNSU

It's good to know there's a performance-accuracy tradeoff. Is the error the same in non-Apple targets? Does replacing FMADD with FNMADD (negative mult) or FNSUB change the magnitude of error in this case? Also, would the relative error be the same if the numbers were random integers or random floats of a given range (-1 to 1; -0.01 to 0.01; -100 to 100; 0 to 1) rather than integers in sequence? ----- In most robotics and AI applications, calculations are unlikely to suffer greatly from inaccuracy but benefit from speed. What are the applications where someone would require floating point math AND both extreme speed and extreme accuracy?

yokaze 5y ago
> What are the applications where someone would require floating point math AND both extreme speed and extreme accuracy? Scientific computation. I'm not even sure, it is a trade-off between (computational) speed and accuracy. In my experience, error-analysis was a mandatory course for physics, but not in robotics. The error analysis was manual and time-consuming. In robotics, no one cared why it occasionally produced NaNs, etc.. people just added a small number here and there. That's faster to program. To put it more positively, I think, the view here is more having an application, which is robust in the face of errors / edge cases. What counts is the output (action). While in science the correct model is part of the output. Edge cases are an important part of it.
newpavlov 5y ago
>In most robotics and AI applications, calculations are unlikely to suffer greatly from inaccuracy but benefit from speed. FMA instructions are more accurate and faster compared to a sequence of separate multiply and add/sub instructions, not less. The article literally talks about it in the beginning. The issue is reproducibility, which with floats can be quite bad even without FMA. Also both robotics and neural networks (I dislike the tendency to call it "AI") can be quite susceptible to errors. Gradient-based optimization algorithms can suffer noticeably from calculation errors. This is why you usually want to train ANNs using 64-bit floats.
lokedhs 5y ago
It's important in finance. Not so much because the last decimals are really important, but when they perform reconciliation they want to ensure that numbers remain consistent over multiple runs, and if one machine returns different values it comes up as errors when the reports are compared, which can trigger all sorts of processes that can be very costly.
- borramakot 5y ago
  The amount of effort and extra hardware required to maintain this reproducibility across different hardware (cpu generations, GPU hardware/drivers, etc) feels insane to me. For example, people seem extremely interested in performance, but insist on using the much slower modes in MKL to maintain the bottom few bits of their golden data. Do you find this genuinely important? If not, have you had any success encouraging e.g. validating against golden data in a non-bitwise comparison?
  - lokedhs 5y ago
    I personally don't find it important. And I have seen that the demand of having numbers match exactly comes from a misunderstanding of how floating point calculations work. Sometimes this is understood by the banks and the solution is to implement a tolerance in the comparisons. In other cases other solutions may have to be used, such as forcing certain reports to be computed on certain software/hardware.
    - perl4ever 5y ago
      >I personally don't find it important You remind me of the alleged UK Post Office scandal: https://www.bbc.com/news/business-56718036 https://www.bbc.com/news/business-56718036
      - lokedhs 5y ago
        That's not what I'm talking about. Floating point computations happen when doing risk calculations, for example computing risk numbers in a Black-Sholes calculation. A difference in the 15'th decimal has no impact on the risk number.
acdha 5y ago
I encountered this in a scientific computing context where a Monte Carlo simulation was designed to give reproducible results across runs. I've forgotten which architectures (Power, Alpha?) hit this but basically some fraction of the test suite failed because the same starting conditions gave slightly different results due to FMADD using a greater precision internal representation before the final round, which meant that over time the particles being simulated would be in different positions (very close, of course).

yokaze 5y ago

> What are the applications where someone would require floating point math AND both extreme speed and extreme accuracy? Scientific computation. I'm not even sure, it is a trade-off between (computational) speed and accuracy. In my experience, error-analysis was a mandatory course for physics, but not in robotics. The error analysis was manual and time-consuming. In robotics, no one cared why it occasionally produced NaNs, etc.. people just added a small number here and there. That's faster to program. To put it more positively, I think, the view here is more having an application, which is robust in the face of errors / edge cases. What counts is the output (action). While in science the correct model is part of the output. Edge cases are an important part of it.

newpavlov 5y ago

>In most robotics and AI applications, calculations are unlikely to suffer greatly from inaccuracy but benefit from speed. FMA instructions are more accurate and faster compared to a sequence of separate multiply and add/sub instructions, not less. The article literally talks about it in the beginning. The issue is reproducibility, which with floats can be quite bad even without FMA. Also both robotics and neural networks (I dislike the tendency to call it "AI") can be quite susceptible to errors. Gradient-based optimization algorithms can suffer noticeably from calculation errors. This is why you usually want to train ANNs using 64-bit floats.

lokedhs 5y ago

It's important in finance. Not so much because the last decimals are really important, but when they perform reconciliation they want to ensure that numbers remain consistent over multiple runs, and if one machine returns different values it comes up as errors when the reports are compared, which can trigger all sorts of processes that can be very costly.

borramakot 5y ago
The amount of effort and extra hardware required to maintain this reproducibility across different hardware (cpu generations, GPU hardware/drivers, etc) feels insane to me. For example, people seem extremely interested in performance, but insist on using the much slower modes in MKL to maintain the bottom few bits of their golden data. Do you find this genuinely important? If not, have you had any success encouraging e.g. validating against golden data in a non-bitwise comparison?
- lokedhs 5y ago
  I personally don't find it important. And I have seen that the demand of having numbers match exactly comes from a misunderstanding of how floating point calculations work. Sometimes this is understood by the banks and the solution is to implement a tolerance in the comparisons. In other cases other solutions may have to be used, such as forcing certain reports to be computed on certain software/hardware.
  - perl4ever 5y ago
    >I personally don't find it important You remind me of the alleged UK Post Office scandal: https://www.bbc.com/news/business-56718036 https://www.bbc.com/news/business-56718036
    - lokedhs 5y ago
      That's not what I'm talking about. Floating point computations happen when doing risk calculations, for example computing risk numbers in a Black-Sholes calculation. A difference in the 15'th decimal has no impact on the risk number.

acdha 5y ago

I encountered this in a scientific computing context where a Monte Carlo simulation was designed to give reproducible results across runs. I've forgotten which architectures (Power, Alpha?) hit this but basically some fraction of the test suite failed because the same starting conditions gave slightly different results due to FMADD using a greater precision internal representation before the final round, which meant that over time the particles being simulated would be in different positions (very close, of course).