3 ms·
Most benchmarks in [0] use LSTMs. [0] https://nlpprogress.com https://nlpprogress.com
by stochastic_monk 8y ago
Most benchmarks in [0] use LSTMs.
[0] https://nlpprogress.com https://nlpprogress.com
- gwern 8y agoNot sure how up to date that is. How many of those things currently listed as SOTA have been compared to, say, BERT baselines?
- carbocation 8y agoUnder their language modeling section, they include the Transformer and have listed some papers from 2019, so it seems to be reasonably up to date.
- stochastic_monk 8y agoFurthermore. transformer xl extends the convolutional architecture to use some recurrence. I think that claims that RNNs have been wholesale replaced by convolutions are premature.
- carbocation 8y agoI can't speak for 'gwern, but I was also under the impression that CNNs had basically replaced RNNs for these tasks. I think my view is colored by Jeremy Howard's FastAI talks. Since he isn't big on publishing papers (though that is changing a bit via collaborations with his students), it may be true that the state of the art is no longer best represented by published papers. (I would argue that avoiding this ambiguity is an argument in favor of publication.)
- stochastic_monk 8y agoIt’s common to use CNNs in practice because they get you most of the way and are so much cheaper to train/run. They are, however, limited in context to the width of the receptive field.
- phowon 8y agoULMFiT is LSTM-based. https://arxiv.org/pdf/1801.06146.pdf https://arxiv.org/pdf/1801.06146.pdf
- carbocation 8y ago(You are right! I'm going to be rewatching the lectures this spring, and it sounds like I should take notes this time.)