4 ms·
It would seem that the best practical approach is to use RNNT as it still lets you do streaming predictions (while Attention won't really let that).
by tasubotadas 7y ago
It would seem that the best practical approach is to use RNNT as it still lets you do streaming predictions (while Attention won't really let that).
- mikaelphi 7y agoAuthor here! RNNT's is indeed a good approach. RNNT's with masked attention are also capable of streaming as well.
- albertzeyer 7y agoIf you need streaming, then yes, RNNT is a good option. If not, encoder-decoder-attention performs a bit better than RNN-T. Note that there are also approaches for encoder-decoder-attention to make that streaming capable, e.g. MoChA or hard attention, etc. Google uses RNN-T on-device. But they are researching on extending it with another encoder-decoder-attention model on-top, to get better performance. This is a quite active research area, and it has not really settled. But CTC is not really too much relevant anymore, as RNNT is just a better variant.
- woodson 7y agoRecent work on transformer transducers with limited right (and left) context seem to give decent results as well: https://arxiv.org/abs/2002.02562 https://arxiv.org/abs/2002.02562
- p1esk 7y agoI opened the pdf, did ctrl+F for 'github', got zero results. Have you reproduced their "decent results"?
- woodson 7y agoESPnet apparently already has implementations of versions of transformer transducers with RNN-T loss (though, with different network architecture). At least the paper cites results on a freely available dataset, right?
- p1esk 7y agoWhy would you not expect Google to post their code? E.g. the transformer paper had it: https://arxiv.org/abs/1706.03762 https://arxiv.org/abs/1706.03762