3 ms·

This is an awesome article! One of my favorite SO answers I remember researching dealt with the Padé approximation of tanh[1] (which I found was significantly b

by dvt 6y ago

This is an awesome article! One of my favorite SO answers I remember researching dealt with the Padé approximation of tanh[1] (which I found was significantly better than the Taylor expansion), but the caveat was that it only worked within a very narrow neighborhood. I will say that the article didn't really touch on techniques that minimize IEEE 754 subnormal[2] performance impact, which is a very interesting problem in itself. A lot of real-life implementations of something like e^x will have various clever ways to avoid subnormals. [1] https://stackoverflow.com/a/6118100/243613 https://stackoverflow.com/a/6118100/243613 [2] https://mathworld.wolfram.com/SubnormalNumber.html https://mathworld.wolfram.com/SubnormalNumber.html

m4r35n357 6y ago
Just a suggestion, have you looked at the MPFR project, and derivatives: https://www.mpfr.org/ https://www.mpfr.org/ http://mpfr.loria.fr/mpfr-current/mpfr.html#index-mpfr_005fsubnormalize http://mpfr.loria.fr/mpfr-current/mpfr.html#index-mpfr_005fs... It is way to deep in the grubby details for my pay grade, but who knows?
fractionalhare 6y ago
Hi, I am the author. You make a good point about subnormals, thank you. I will jot that down for a future update to the doc :) It is funny you mention the Padé approximation; I was debating whether or not to include that in this article. Originally I planned to stop after Minimax (using Remez), but if I include rational approximation schemes anyway (e.g. Carathéodory-Fejer), it probably makes sense to implement and analyze that as well.
- dia80 6y ago
  Just jumping into to say I am loving the article, thanks! I'm dealing with a lot of exps in an optimization problem right now and frankly only have cursory understanding of what's going on under the hood so this was very enlightening for me. Thanks again.
raphlinus 6y ago
You might enjoy a few of my favorite sigmoids: https://raphlinus.github.io/audio/2018/09/05/sigmoid.html https://raphlinus.github.io/audio/2018/09/05/sigmoid.html That's not a deep exploration of the numerical analysis (especially compared with OP), but it does make one point: traditional "numerical analysis" techniques tend to assume the traditional four mathematical operators as canonical, but if your goal is to get something computed on actual computer hardware quickly and reasonably accurately, you have a wider palette to play with. In particular, inverse square root is a primitive about as efficient as reciprocal on modern hardware (or if using Carmack's wtf trick).
- dvt 6y ago
  This is great! Sometimes I wish I took more analysis back in school.

by dvt 6y ago

m4r35n357 6y ago

Just a suggestion, have you looked at the MPFR project, and derivatives: https://www.mpfr.org/ https://www.mpfr.org/ http://mpfr.loria.fr/mpfr-current/mpfr.html#index-mpfr_005fsubnormalize http://mpfr.loria.fr/mpfr-current/mpfr.html#index-mpfr_005fs... It is way to deep in the grubby details for my pay grade, but who knows?

fractionalhare 6y ago

Hi, I am the author. You make a good point about subnormals, thank you. I will jot that down for a future update to the doc :) It is funny you mention the Padé approximation; I was debating whether or not to include that in this article. Originally I planned to stop after Minimax (using Remez), but if I include rational approximation schemes anyway (e.g. Carathéodory-Fejer), it probably makes sense to implement and analyze that as well.

dia80 6y ago
Just jumping into to say I am loving the article, thanks! I'm dealing with a lot of exps in an optimization problem right now and frankly only have cursory understanding of what's going on under the hood so this was very enlightening for me. Thanks again.

raphlinus 6y ago

You might enjoy a few of my favorite sigmoids: https://raphlinus.github.io/audio/2018/09/05/sigmoid.html https://raphlinus.github.io/audio/2018/09/05/sigmoid.html That's not a deep exploration of the numerical analysis (especially compared with OP), but it does make one point: traditional "numerical analysis" techniques tend to assume the traditional four mathematical operators as canonical, but if your goal is to get something computed on actual computer hardware quickly and reasonably accurately, you have a wider palette to play with. In particular, inverse square root is a primitive about as efficient as reciprocal on modern hardware (or if using Carmack's wtf trick).

dvt 6y ago
This is great! Sometimes I wish I took more analysis back in school.