> learn gradient descent by gradient descent. As far as 'learning gradient descent by gradient descent' goes, I think it's still an open question if a tiny RNN

> learn gradient descent by gradient descent. As far as 'learning gradient descent by gradient descent' goes, I think it's still an open question if a tiny RNN actually can improve meaningfully over ADAM etc :) I don't recall the paper showing any wallclock times, which suggested to me that the RNN was way slower even if it trained faster. The individual parameter adjustments are so low in the hierarchy of actions that the value may be minimal compared to higher up like architecture design.