3 ms·
I'm unclear about why z needs to be parameterized in terms of two functions, one unbounded, u(x), and a one bounded, tanh(g(x,h)). Shouldn't the squashing funct
by unixpickle 10y ago
I'm unclear about why z needs to be parameterized in terms of two functions, one unbounded, u(x), and a one bounded, tanh(g(x,h)). Shouldn't the squashing function f() prevent explosion anyway? And how would this parameterization prevent vanishing gradients?
- jostmey 10y agoGood question! In retrospect, I should have explained that section differently. I will probably revise that section of the text (that's what I love about arxiv). We don't want the recurrent terms "h_{t-1}" to dominate the output of "z". But we don't want to get eliminate the recurrent terms because without "h_{t-1}" the output of "z" will not be sequence dependent. So the "tanh" term is included to allow the recurrent terms to change the sign of "z". I don't know if that helps. Thanks!
- unixpickle 10y agoAh, now it's much more clear. By the way, have you tested the model without parameterizing z the way you did? Different z's might be worth further exploration.
- jostmey 10y agoYes, different models for "z" are worth exploring! You should try different forms and see what works!