4 ms·
Sounds promising! A couple of questions after a quick read, but sure I have more later: Is it faster than LSTM for training / inference? and, are you planning t
by jorgemf 10y ago
Sounds promising! A couple of questions after a quick read, but sure I have more later: Is it faster than LSTM for training / inference? and, are you planning to use it for more complex problems as translation or image to caption?
- jostmey 10y agoThe RWA model requires fewer training steps than a LSTM model on the tasks I tested (https://arxiv.org/pdf/1703.01253.pdf https://arxiv.org/pdf/1703.01253.pdf). Even better, the RWA model uses over 25% fewer parameters (because it doesn't have lots of gates like a LSTM model). So the RWA uses less computer memory than a LSTM model. To answer your second question: Yes I do plan on testing it on more challenging problems! I just need enough time and the GPU rig.
- jorgemf 10y agoI was wondering the time per step, I am aware it converges faster. Let me know when you have more results! I want to start doing things with text soon and I would love to try RWA
- jostmey 10y agoThe RWA model is described in equations (7) of this paper: https://arxiv.org/pdf/1703.01253.pdf https://arxiv.org/pdf/1703.01253.pdf I think you will see that the cost per training set is just as cheap if not cheaper than a LSTM model. On my machine, the RWA model completes each training step faster than a LSTM model. I don't know if that will hold for GPUs.
- singularity2001 10y agoYes, give us wall time please
- deleted 10y ago[deleted]
- kastnerkyle 10y agoSo in that case it has the same amount of parameters as GRU - did you compare with that?
- jostmey 10y agoNo, I have not tried the GRU model