3 ms·
Why did he not used Time Convolutional Networks? [1] https://openreview.net/pdf?id=rk8wKk-R- https://openreview.net/pdf?id=rk8wKk-R- [1] shows that TCN may ou
by thesz 8y ago
Why did he not used Time Convolutional Networks?
[1] https://openreview.net/pdf?id=rk8wKk-R- https://openreview.net/pdf?id=rk8wKk-R-
[1] shows that TCN may outperform LSTM on tasks that really needs memory.
- 317070 8y agoBecause LSTM's are proven over time to be industry grade, and are not latest-greatest-but-no-one-has-really-reproduced-the-results-yet. A paper with inductive results is just a first step in a really long process until ideas are commonplace accepted.
- thesz 8y agoFor every time you say "LSTM is industry grade" I would point you to large companies (Google-level) which train their acoustic models with TCN architectures. He used Transformer, why not TCN?
- Cybiote 8y agoIt's rare that a model introduced in a paper has fair comparisons. You'll want to see it used by many different authors before one can speak with confidence on claimed improvements. In this paper, Bellard makes a comparison to the Transformer, some variant of which holds most NLP records. Of note is that in this test, the Transformer did not outperform the LSTM. Is it that Transformers need to be larger to outperform or LSTMs are more suited to compression or that more epochs are required? This deserves further investigation, what are the required conditions for Transformers to underperform LSTMs?
- thesz 8y agoThe problem in the paper is to predict most probable "words". In NLP terms, model must be optimized for perplexity. Transformer holds records not for perplexity (it is too heavy for that) but for other tasks like machine translation or semantic similarity, I believe.