3 ms·
What's the reasoning behind choosing LSTMs here? Won't an attention model be even more effective?
by melony 4y ago
What's the reasoning behind choosing LSTMs here? Won't an attention model be even more effective?
- woadwarrior01 4y agoDid you see who the co-author of the paper is? :)
- Skyy93 4y agoAs you should know: Schmidhuber even invented the Transformer-Models, to be read in: https://arxiv.org/abs/2102.11174 https://arxiv.org/abs/2102.11174 ;)
- YorkshireSeason 4y agoThe claim of formal equivalence is restricted to "linear transformers". The tone of the comment seems to suggest that S.'s claim is slightly ridiculous. Can you point out a concrete technical shortcoming of the paper?
- Skyy93 4y agoNaaa, I just wanted to pointing out his attitude towards other research.
- YorkshireSeason 4y agoIt is valuable to point out connections with existing work, if only to avoid reinventing the wheel, and properly stand on the shoulders of gigants: אֵין כָּל חָדָשׁ תַּחַת הַשָּׁמֶשׁ (there is nothing new under the sun).
- Skyy93 4y agoYep, but you can do this without acting like an ass.
- melony 4y agoI bet OP was his student
- YorkshireSeason 4y agoHow much are you willing to bet?
- mlajtos 4y agoMy guess is that tiny LSTM is less computationally expensive than a tiny transformer. Since paper proposes replacing every weight with this tiny shared net, you want to be as tiny & performant as possible.
- lettergram 4y agoTransformers are more complicated than RNNs and require more fine tuning. I’m guessing the RNNs were used to simplify the problem. I’m not even sure transformers would work here given we’re theyre dropping them into a process
- laichzeit0 4y agoIf you want it tiny and performant and choose LSTM over transformers why not just skip LSTM and use GRUs?