3 ms·

You can very likely compress the table and/or interpolate between entries. It's not like that those entries are totally random.

• AstralStorm 6y ago
  You can actually do that as well with the division itself. But you need a kind of fancy interpolation, I forget the name, used for rational functions. Polynomials won't do. Edit: I think this is either Restrictive Pade's approximation or barycentric rational interpolation. Or both combined. Normally instead division is approximated with Newton-Raphson iteration. I think 6 steps is enough for 16 bit.
  • fuzzfactor 6y ago
    It's a simple linear function, and the delta Y has enough digits to overcome significant rounding error in comparison to measurement error. In this application no rounding is needed either since the difference between rounding and truncation is also not significant. You don't need the computer to do any of your multiplication or store any table. You can start from a sufficiently low reference point, then add delta Y as you increment X. Xref + deltaX = Yref + deltaY