4 ms·
The stuff in Appendix C worries me a little. Dividing the numerator and denominator indeed does not change the output at timestep t. However, if you use the new
by unixpickle 10y ago
The stuff in Appendix C worries me a little. Dividing the numerator and denominator indeed does not change the output at timestep t. However, if you use the new, smaller numerator and denominator at later timesteps, later outputs will be affected.
Suppose, for example, that you divide the numerator and denominator by 2, giving n/2 and d/2. In the next timestep, suppose a<0. You now do n=n/2+za and d=d/2+a. What you really ought to be doing is (n+za)/2 and (d+a)/2. In other words, once you scale n and d by some factor, all later timesteps should have that same factor thrown in (in order to preserve the outputs from the theoretical model).
If you do things the way I suspect, then any positive value of `a` essentially gives more priority to later timesteps (since their numerators and denominators are not scaled down by the theoretically necessary amount). This seems to defeat most of the purpose of the model.
Edit: by the way, I tried looking through your code to figure out what it was actually doing. However, it does seem like you are doing what I thought. If I am correct, then your numerical stability code does actually affect the model's output.
- jostmey 10y agoOh no! I think you may be right. It is possible to correct the model for numerical stability in a manner that doesn't impact the output, but my approach fails at that. The correction from the previous step needs to be saved and carried over to the next step, just like the numerator and denominator are saved and carried over. I think I will have to re-run all the results once I fix the code. This is a bummer -- Thanks for finding this! Jared