3 ms·

Hi! > I guess you prefer an "end-to-end" model over a hybrid HMM/NN model, for simplicity, right? Simplicity and ease of targeting other languages, yes. We're

Hi! > I guess you prefer an "end-to-end" model over a hybrid HMM/NN model, for simplicity, right? Simplicity and ease of targeting other languages, yes. We're a small team. > As far as I remember, you use CTC, right? I always wondered why you have chosen CTC, and not some better model, like RNN-T, RNA, or some of the streaming attention variants. We started DeepSpeech in 2016, before these recent developments for end-to-end ASR were mainstream/SotA. > Or is this simply because there currently is no simple ready-to-use implementation for those? Implementing the model architecture for training is only part of the problem for us. We have a hand-crafted inference graph to allow for small and efficient client code and inference models, and the more complex the architecture is, the trickier it gets to make sure it all works on all platforms, including TFLite, with quantization, etc. We're investigating alternative architectures as well as mixed CTC/RNN LMs to deal with language model size, but no final decisions made yet. > Note that we published some TF code recently for some streaming attention variants, and plan to publish some RNN-T/RNA code soon. Nice! Can you share a link to the streaming attention code?

mostlyjason 7y ago
I really hope you adopt the latest models particularly streaming attention variants. I think you should validate with users the assumption that latency is more important than WER. IMHO the WER is more important than latency improvements in the millisecond range. The most frustrating thing is having to dictate over and over and the transcription is incorrect each time. Consider that the time to a correct transcription is the latency plus error correction. If error correction is manual it will be orders of magnitude slower, so optimize for WER. I’m terms of competition, Siri has latency in the 5+ second range due to the network call especially in area with poor data rates. I think a client side model like yours will easily win in this category. If you’re already ahead here, why not focus on WER next? Another great capability is to generate alternative transcriptions for words with low confidence values to allow for quick error correction. Do you offer something like this today? Also, consider the long term view that new models are constantly being released and refined. It’d be best to have an architecture that allows quick replacement without a lot of hand tuning, or where the tuning can be automated to a greater extent.
- whyleyc 7y ago
  Thanks for all the hard work you have put in so far @reubenmorais +1000 to @mostlyjason's comment - Great latency figures mean nothing if the word error rate is high, since it dents confidence in the output (so why use DeepSpeech?) and (as the parent comment notes) necessitates manual error correction. I would love to see a future release focus on optimizing WER for these reasons.
albertzeyer 7y ago
> Nice! Can you share a link to the streaming attention code? It uses our TensorFlow framework Returnn (https://github.com/rwth-i6/returnn https://github.com/rwth-i6/returnn). We currently only have some configs/code online for hard attention variants, or segmental models. They can be found here: https://github.com/rwth-i6/returnn-experiments/tree/master/2019-monotonic-latent-attention https://github.com/rwth-i6/returnn-experiments/tree/master/2... Note that the configs are maybe not so cleaned up, as this is very much research. This is for a paper we submitted to ICASSP. We did not publish the paper yet elsewhere, but I can send you a copy by mail (just contact me: albzey@gmail.com). Also, as this is research focused, the encoder here is also a BLSTM, because we wanted to compare this work to other global soft attention models, and have the comparison mostly focused on the streaming attention modeling aspect. And also there is some lookahead, which is currently unlimited. So it would need a few modifications to really be used for online streaming. I'm also not sure whether this is the best model, or whether some of the many other variants (MoChA, RNN-T, etc) are maybe better. Edit: I forgot, we also have some local windowed attention variants, which can also be applied for streaming: https://github.com/rwth-i6/returnn-experiments/tree/master/2019-asr-local-attention https://github.com/rwth-i6/returnn-experiments/tree/master/2...

mostlyjason 7y ago

I really hope you adopt the latest models particularly streaming attention variants. I think you should validate with users the assumption that latency is more important than WER. IMHO the WER is more important than latency improvements in the millisecond range. The most frustrating thing is having to dictate over and over and the transcription is incorrect each time. Consider that the time to a correct transcription is the latency plus error correction. If error correction is manual it will be orders of magnitude slower, so optimize for WER. I’m terms of competition, Siri has latency in the 5+ second range due to the network call especially in area with poor data rates. I think a client side model like yours will easily win in this category. If you’re already ahead here, why not focus on WER next? Another great capability is to generate alternative transcriptions for words with low confidence values to allow for quick error correction. Do you offer something like this today? Also, consider the long term view that new models are constantly being released and refined. It’d be best to have an architecture that allows quick replacement without a lot of hand tuning, or where the tuning can be automated to a greater extent.

whyleyc 7y ago
Thanks for all the hard work you have put in so far @reubenmorais +1000 to @mostlyjason's comment - Great latency figures mean nothing if the word error rate is high, since it dents confidence in the output (so why use DeepSpeech?) and (as the parent comment notes) necessitates manual error correction. I would love to see a future release focus on optimizing WER for these reasons.

albertzeyer 7y ago

> Nice! Can you share a link to the streaming attention code? It uses our TensorFlow framework Returnn (https://github.com/rwth-i6/returnn https://github.com/rwth-i6/returnn). We currently only have some configs/code online for hard attention variants, or segmental models. They can be found here: https://github.com/rwth-i6/returnn-experiments/tree/master/2019-monotonic-latent-attention https://github.com/rwth-i6/returnn-experiments/tree/master/2... Note that the configs are maybe not so cleaned up, as this is very much research. This is for a paper we submitted to ICASSP. We did not publish the paper yet elsewhere, but I can send you a copy by mail (just contact me: albzey@gmail.com). Also, as this is research focused, the encoder here is also a BLSTM, because we wanted to compare this work to other global soft attention models, and have the comparison mostly focused on the streaming attention modeling aspect. And also there is some lookahead, which is currently unlimited. So it would need a few modifications to really be used for online streaming. I'm also not sure whether this is the best model, or whether some of the many other variants (MoChA, RNN-T, etc) are maybe better. Edit: I forgot, we also have some local windowed attention variants, which can also be applied for streaming: https://github.com/rwth-i6/returnn-experiments/tree/master/2019-asr-local-attention https://github.com/rwth-i6/returnn-experiments/tree/master/2...