4 ms·
Furthermore. transformer xl extends the convolutional architecture to use some recurrence. I think that claims that RNNs have been wholesale replaced by convolu
by stochastic_monk 8y ago
Furthermore. transformer xl extends the convolutional architecture to use some recurrence. I think that claims that RNNs have been wholesale replaced by convolutions are premature.
- carbocation 8y agoI can't speak for 'gwern, but I was also under the impression that CNNs had basically replaced RNNs for these tasks. I think my view is colored by Jeremy Howard's FastAI talks. Since he isn't big on publishing papers (though that is changing a bit via collaborations with his students), it may be true that the state of the art is no longer best represented by published papers. (I would argue that avoiding this ambiguity is an argument in favor of publication.)
- stochastic_monk 8y agoIt’s common to use CNNs in practice because they get you most of the way and are so much cheaper to train/run. They are, however, limited in context to the width of the receptive field.
- phowon 8y agoULMFiT is LSTM-based. https://arxiv.org/pdf/1801.06146.pdf https://arxiv.org/pdf/1801.06146.pdf
- carbocation 8y ago(You are right! I'm going to be rewatching the lectures this spring, and it sounds like I should take notes this time.)