5 ms·
This seems to be a CTC model. CTC is not really the best option for a good end-to-end system. Encoder-decoder-attention models or RNN-T models are both better a
by albertzeyer 7y ago
This seems to be a CTC model. CTC is not really the best option for a good end-to-end system. Encoder-decoder-attention models or RNN-T models are both better alternatives.
There is also not really a problem about available open source code. There are countless of open source projects which already have that mostly ready to use, for all the common DL frameworks, like TF, PyTorch, Jax, MXNet, whatever. For anyone with a bit of ML experience, this should really not be too hard to setup.
But then to get good performance, on your own dataset, what you really need is experience. Probably taking some existing pipeline will get you some model, with an okish word-error-rate. But then you should tune it. In any case, even without tuning, probably encoder-decoder-attention models will perform better than CTC models.
- option 7y agoI question that you need full attention in the acoustic model. The pronunciation of a word in the middle of phrase does not have much dependence on the beginning. You do need attention in the language model part of the pipeline
- albertzeyer 7y agoAlmost all the literature which compares CTC and encoder-decoder-attention models shows pretty well that encoder-decoder-attention performs better than CTC in the acoustic model. See for example here as an overview (my own work, already a bit outdated, but attention has even improved much more since then): https://openreview.net/pdf?id=S1gp9v_jsm https://openreview.net/pdf?id=S1gp9v_jsm
- dylanbfox 7y agoThat's right - most literature does show that encoder-decoder architectures outperform CTC. I think one of the main reasons for this is that CTC assumes the label outputs are conditionally independent of each other, which is a pretty big flaw in that loss function. The blog does mention Listen-Attend-Spell (which is an encoder-decoder architecture) as an alternative to the CTC model.
- tasubotadas 7y agoIt would seem that the best practical approach is to use RNNT as it still lets you do streaming predictions (while Attention won't really let that).
- mikaelphi 7y agoAuthor here! RNNT's is indeed a good approach. RNNT's with masked attention are also capable of streaming as well.
- albertzeyer 7y agoIf you need streaming, then yes, RNNT is a good option. If not, encoder-decoder-attention performs a bit better than RNN-T. Note that there are also approaches for encoder-decoder-attention to make that streaming capable, e.g. MoChA or hard attention, etc. Google uses RNN-T on-device. But they are researching on extending it with another encoder-decoder-attention model on-top, to get better performance. This is a quite active research area, and it has not really settled. But CTC is not really too much relevant anymore, as RNNT is just a better variant.
- woodson 7y agoRecent work on transformer transducers with limited right (and left) context seem to give decent results as well: https://arxiv.org/abs/2002.02562 https://arxiv.org/abs/2002.02562
- p1esk 7y agoI opened the pdf, did ctrl+F for 'github', got zero results. Have you reproduced their "decent results"?
- woodson 7y agoESPnet apparently already has implementations of versions of transformer transducers with RNN-T loss (though, with different network architecture). At least the paper cites results on a freely available dataset, right?
- bginsburg 7y agoActually end-2-end CTC models are very good - Wav2letter, Jasper, QuartzNet - all these models are much better than DeepSpeech2.
- mikaelphi 7y agoAuthor here! CTC models perform quite well and are easy to get started for beginners with an added benefit of real-time streaming capabilities. RNNT's and Encoder-Decoder like Listen-attend-spell are also very solid choices, and literature points to slightly higher accuracy with them on academic datasets. RNNT's being and extension of CTC are streamable as well.
- woodson 7y agoAgreed, and that's where it seems having lots of experience working with speech data helps more than trying to brute-force it with just larger CTC models and "more" data of dubious quality.