4 ms·

Author here. I would add to what the sibling comments have mentioned by saying that SotA results should be taken with a grain of salt. Our engine is capable of

Author here. I would add to what the sibling comments have mentioned by saying that SotA results should be taken with a grain of salt. Our engine is capable of streaming (processing the audio as it's being recorded), which is not doable with architectures that have bidirectional decoders or attention mechanisms that require the whole encoder input ahead of time. For real world applications, this is absolutely crucial, users want latency numbers on the order of milliseconds, not seconds. This is why, if you run a standard test set like LibriSpeech on, say, a commercial offering from Google, it will perform considerably worse than state of the art according to Google papers. This repository [0] has a benchmark of some commercial offerings. Our model beats all of those on Librispeech clean and other (except for Speechmatics on Librispeech clean), as well as on Common Voice. But note that the Common Voice corpus used in that benchmark is very old. In sum, I would compare this against solutions that go for the same space: fast, client-side ASR, rather than state of the art. [0] https://github.com/Franck-Dernoncourt/ASR_benchmark#benchmark-results https://github.com/Franck-Dernoncourt/ASR_benchmark#benchmar...

albertzeyer 7y ago
Hi! I'm from the RWTH team (https://paperswithcode.com/sota/speech-recognition-on-librispeech-test-clean https://paperswithcode.com/sota/speech-recognition-on-libris...). Our best system (2.3% WER) (trained only on the Librispeech data, i.e. much less data than your model) currently is a hybrid HMM/NN model, and you are right, the acoustic model uses a BLSTM. However, we have shown in other work (https://www-i6.informatik.rwth-aachen.de/publications/download/1009/Zeyer--2016.pdf https://www-i6.informatik.rwth-aachen.de/publications/downlo...) that you can expect similar performance in an online system, or not much worse (there is also many related work, which often is maybe 5-10% relative worse than the offline system). I guess you prefer an "end-to-end" model over a hybrid HMM/NN model, for simplicity, right? As far as I remember, you use CTC, right? I always wondered why you have chosen CTC, and not some better model, like RNN-T, RNA, or some of the streaming attention variants. They should give you all the same properties (online capable, low latency, simple, end-to-end), but much better WER performance. Or is this simply because there currently is no simple ready-to-use implementation for those? Note that we published some TF code recently for some streaming attention variants, and plan to publish some RNN-T/RNA code soon.
- reubenmorais 7y ago
  Hi! > I guess you prefer an "end-to-end" model over a hybrid HMM/NN model, for simplicity, right? Simplicity and ease of targeting other languages, yes. We're a small team. > As far as I remember, you use CTC, right? I always wondered why you have chosen CTC, and not some better model, like RNN-T, RNA, or some of the streaming attention variants. We started DeepSpeech in 2016, before these recent developments for end-to-end ASR were mainstream/SotA. > Or is this simply because there currently is no simple ready-to-use implementation for those? Implementing the model architecture for training is only part of the problem for us. We have a hand-crafted inference graph to allow for small and efficient client code and inference models, and the more complex the architecture is, the trickier it gets to make sure it all works on all platforms, including TFLite, with quantization, etc. We're investigating alternative architectures as well as mixed CTC/RNN LMs to deal with language model size, but no final decisions made yet. > Note that we published some TF code recently for some streaming attention variants, and plan to publish some RNN-T/RNA code soon. Nice! Can you share a link to the streaming attention code?
  - mostlyjason 7y ago
    I really hope you adopt the latest models particularly streaming attention variants. I think you should validate with users the assumption that latency is more important than WER. IMHO the WER is more important than latency improvements in the millisecond range. The most frustrating thing is having to dictate over and over and the transcription is incorrect each time. Consider that the time to a correct transcription is the latency plus error correction. If error correction is manual it will be orders of magnitude slower, so optimize for WER. I’m terms of competition, Siri has latency in the 5+ second range due to the network call especially in area with poor data rates. I think a client side model like yours will easily win in this category. If you’re already ahead here, why not focus on WER next? Another great capability is to generate alternative transcriptions for words with low confidence values to allow for quick error correction. Do you offer something like this today? Also, consider the long term view that new models are constantly being released and refined. It’d be best to have an architecture that allows quick replacement without a lot of hand tuning, or where the tuning can be automated to a greater extent.
option 7y ago
NVIDIA has QuartzNet which contains only 19M weights and achieves 3.9% on test clean without language model and less then 3 with LM. Code (Pytorch): https://github.com/NVIDIA/NeMo https://github.com/NVIDIA/NeMo Paper: https://arxiv.org/pdf/1910.10261.pdf https://arxiv.org/pdf/1910.10261.pdf
IshKebab 7y ago
> Our engine is capable of streaming A bit misleading given that decoding can't be streamed yet.
- lunixbochs 7y ago
  I’m not sure what you mean. The image in the blog post clearly shows small chunks of decode time happening during the speech (every second).
  - IshKebab 7y ago
    The API documentation says decoding starts from the beginning every time. Could just be the Python API hasn't been updated I guess? Edit: never mind - the API documentation was just out of date.
IshKebab 7y ago
> For real world applications, this is absolutely crucial, users want latency numbers on the order of milliseconds, not seconds. You should follow Google's approach - give fast live results that don't depend on data from the future, but also go back and correct old words when you do have that data. It's kind of how humans work really.
lunixbochs 7y ago
> For real world applications, this is absolutely crucial, users want latency numbers on the order of milliseconds, not seconds. My wav2letter research does not use streaming, despite supporting streaming, because I get lower average latency (roughly 0.02x RTF including decoding, or 80ms for a 4000ms input audio like you describe in the article) by running the entire encoder CNN at once instead of running it in chunks. This is also nice because the CPU usage is practically nothing (0.1%) until you stop talking. I have a web demo here [1], with a relatively terrible language model (mostly just been working on acoustic modeling so far, as it’s just me). Most of the latency is waiting for the JavaScript VAD (which is much slower than webrtcvad and I couldn’t figure out how to tune it) and waiting for the network. If you look at the network inspector, the server should report its encode and decode times. [1] https://web2letter-west-1.talonvoice.com/ https://web2letter-west-1.talonvoice.com/

albertzeyer 7y ago

Hi! I'm from the RWTH team (https://paperswithcode.com/sota/speech-recognition-on-librispeech-test-clean https://paperswithcode.com/sota/speech-recognition-on-libris...). Our best system (2.3% WER) (trained only on the Librispeech data, i.e. much less data than your model) currently is a hybrid HMM/NN model, and you are right, the acoustic model uses a BLSTM. However, we have shown in other work (https://www-i6.informatik.rwth-aachen.de/publications/download/1009/Zeyer--2016.pdf https://www-i6.informatik.rwth-aachen.de/publications/downlo...) that you can expect similar performance in an online system, or not much worse (there is also many related work, which often is maybe 5-10% relative worse than the offline system). I guess you prefer an "end-to-end" model over a hybrid HMM/NN model, for simplicity, right? As far as I remember, you use CTC, right? I always wondered why you have chosen CTC, and not some better model, like RNN-T, RNA, or some of the streaming attention variants. They should give you all the same properties (online capable, low latency, simple, end-to-end), but much better WER performance. Or is this simply because there currently is no simple ready-to-use implementation for those? Note that we published some TF code recently for some streaming attention variants, and plan to publish some RNN-T/RNA code soon.

reubenmorais 7y ago
Hi! > I guess you prefer an "end-to-end" model over a hybrid HMM/NN model, for simplicity, right? Simplicity and ease of targeting other languages, yes. We're a small team. > As far as I remember, you use CTC, right? I always wondered why you have chosen CTC, and not some better model, like RNN-T, RNA, or some of the streaming attention variants. We started DeepSpeech in 2016, before these recent developments for end-to-end ASR were mainstream/SotA. > Or is this simply because there currently is no simple ready-to-use implementation for those? Implementing the model architecture for training is only part of the problem for us. We have a hand-crafted inference graph to allow for small and efficient client code and inference models, and the more complex the architecture is, the trickier it gets to make sure it all works on all platforms, including TFLite, with quantization, etc. We're investigating alternative architectures as well as mixed CTC/RNN LMs to deal with language model size, but no final decisions made yet. > Note that we published some TF code recently for some streaming attention variants, and plan to publish some RNN-T/RNA code soon. Nice! Can you share a link to the streaming attention code?
- mostlyjason 7y ago
  I really hope you adopt the latest models particularly streaming attention variants. I think you should validate with users the assumption that latency is more important than WER. IMHO the WER is more important than latency improvements in the millisecond range. The most frustrating thing is having to dictate over and over and the transcription is incorrect each time. Consider that the time to a correct transcription is the latency plus error correction. If error correction is manual it will be orders of magnitude slower, so optimize for WER. I’m terms of competition, Siri has latency in the 5+ second range due to the network call especially in area with poor data rates. I think a client side model like yours will easily win in this category. If you’re already ahead here, why not focus on WER next? Another great capability is to generate alternative transcriptions for words with low confidence values to allow for quick error correction. Do you offer something like this today? Also, consider the long term view that new models are constantly being released and refined. It’d be best to have an architecture that allows quick replacement without a lot of hand tuning, or where the tuning can be automated to a greater extent.

option 7y ago

NVIDIA has QuartzNet which contains only 19M weights and achieves 3.9% on test clean without language model and less then 3 with LM. Code (Pytorch): https://github.com/NVIDIA/NeMo https://github.com/NVIDIA/NeMo Paper: https://arxiv.org/pdf/1910.10261.pdf https://arxiv.org/pdf/1910.10261.pdf

IshKebab 7y ago

> Our engine is capable of streaming A bit misleading given that decoding can't be streamed yet.

lunixbochs 7y ago
I’m not sure what you mean. The image in the blog post clearly shows small chunks of decode time happening during the speech (every second).
- IshKebab 7y ago
  The API documentation says decoding starts from the beginning every time. Could just be the Python API hasn't been updated I guess? Edit: never mind - the API documentation was just out of date.

IshKebab 7y ago

> For real world applications, this is absolutely crucial, users want latency numbers on the order of milliseconds, not seconds. You should follow Google's approach - give fast live results that don't depend on data from the future, but also go back and correct old words when you do have that data. It's kind of how humans work really.

lunixbochs 7y ago

> For real world applications, this is absolutely crucial, users want latency numbers on the order of milliseconds, not seconds. My wav2letter research does not use streaming, despite supporting streaming, because I get lower average latency (roughly 0.02x RTF including decoding, or 80ms for a 4000ms input audio like you describe in the article) by running the entire encoder CNN at once instead of running it in chunks. This is also nice because the CPU usage is practically nothing (0.1%) until you stop talking. I have a web demo here [1], with a relatively terrible language model (mostly just been working on acoustic modeling so far, as it’s just me). Most of the latency is waiting for the JavaScript VAD (which is much slower than webrtcvad and I couldn’t figure out how to tune it) and waiting for the network. If you look at the network inspector, the server should report its encode and decode times. [1] https://web2letter-west-1.talonvoice.com/ https://web2letter-west-1.talonvoice.com/