I thought everyone knew that softmax (and specifically exp functions in it) are poison. I have always worked around them, for example by using large epsilons (a

I thought everyone knew that softmax (and specifically exp functions in it) are poison. I have always worked around them, for example by using large epsilons (approaching one actually), and using low-order polynomial approximations for the exp functions. I thought everyone does that, because you don't need to work long with these models to get NaNs, and when you check why you see it's because of the exp functions. Then you fix it. Apparently people don't. It's not like the neural models care if you approximate functions. They couldn't care less actually.