3 ms·

Yes, I understood that. However repeated conversions can lead to rounding errors such as catastrophic cancellation. This is even more esoteric than “grams here

Yes, I understood that. However repeated conversions can lead to rounding errors such as catastrophic cancellation. This is even more esoteric than “grams here kilograms there” and thus easier to fall prey to. Depending on how precise your measurements are you could easily drop significant digits or add random noise, especially if you do large scale changes. https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.h...

sbradford26 4y ago
Yeah we tended to utilize fixed point for many of our variables and documented the precision for each one. Then an analysis was done verifying that we were maintaining precision through the calculations from input from a sensor to output.
- fnordpiglet 4y ago
  The problem is almost nothing today accelerates fixed point and everything supports floating point accelerations, which will speed up your ability to calculate the wrong number by many orders of magnitude.
  - sbradford26 4y ago
    Yeah that makes sense. In aerospace they will fall on the side of accuracy even at the expense of speed. But they also tend to utilize FPGAs that can be designed to handle fixed point calculations quickly but with that comes a lot of cost and specialty hardware.
    - fnordpiglet 4y ago
      Another place it makes issues is financial modeling, particularly for complex derivatives where the model might be extremely complex and small errors can accumulate into meaningful errors in risk calculations. Fixed point is sometimes used in finance but performance is also a real concern. There’s a lot you can do to reduce fp errors if your numerical libraries are carefully constructed. But it has often made me wonder why there aren’t processor lines with high performance fixed point since the math is extremely easy - even by just shifting the mantissa and working in integer space then shifting it back.

sbradford26 4y ago

Yeah we tended to utilize fixed point for many of our variables and documented the precision for each one. Then an analysis was done verifying that we were maintaining precision through the calculations from input from a sensor to output.

fnordpiglet 4y ago
The problem is almost nothing today accelerates fixed point and everything supports floating point accelerations, which will speed up your ability to calculate the wrong number by many orders of magnitude.
- sbradford26 4y ago
  Yeah that makes sense. In aerospace they will fall on the side of accuracy even at the expense of speed. But they also tend to utilize FPGAs that can be designed to handle fixed point calculations quickly but with that comes a lot of cost and specialty hardware.
  - fnordpiglet 4y ago
    Another place it makes issues is financial modeling, particularly for complex derivatives where the model might be extremely complex and small errors can accumulate into meaningful errors in risk calculations. Fixed point is sometimes used in finance but performance is also a real concern. There’s a lot you can do to reduce fp errors if your numerical libraries are carefully constructed. But it has often made me wonder why there aren’t processor lines with high performance fixed point since the math is extremely easy - even by just shifting the mantissa and working in integer space then shifting it back.