4 ms·
Since the authors appear to be listening on here, I have a question about the method. From my experience, RNN weights and the recurrent weights Tf in an LSTM t
by mgraczyk 10y ago
Since the authors appear to be listening on here, I have a question about the method.
From my experience, RNN weights and the recurrent weights Tf in an LSTM tend to look more like (I + low_rank) rather than low_rank. To be more specific, I gather that with your F-LSTM you do:
T1 = W1 * input
T2 = W2 * T1
output = T2
so
output = W2 * W1 * input
Where W1 and W2 are "factorized by design".
However, it seems like the recurrent weights (those for f in your paper) should look more like
T1 = W1 * input
T2 = W2 * T1
output = input + T2
so
output = (I + W2 * W1) * input
That way, you are imposing the simplification that the Tf ~= (I + low_rank) instead of Tf ~= low_rank.
Have you considered this?
- option 10y ago>>"RNN weights and the recurrent weights Tf in an LSTM tend to look more like (I + low_rank)" - Do you have reference for this? But thanks for suggestion, I'll have a look into this. >>"I gather that with your F-LSTM you do:" - Your understanding of F-LSTM looks correct >>"output = input + T2" Not quite clear where to put non-linearities. But this looks similar to residual connections. Which, in my experience, is almost always a good idea.
- juxtaposicion 10y agoIf you've seen tensor trains, might be interesting to reduce the rank even further by splitting an input into many sequential small matrices (instead of just two.)