5 ms·
> It achieves a 7.5% word error rate on the LibriSpeech test clean benchmark Anyone have a comparison for how good/bad that is compared to other solutions, and
by detaro 7y ago
> It achieves a 7.5% word error rate on the LibriSpeech test clean benchmark
Anyone have a comparison for how good/bad that is compared to other solutions, and what it means for practical usage, if that can be guessed at from a single number?
- Traktor 7y agoYou can find a leaderboard here: https://paperswithcode.com/sota/speech-recognition-on-librispeech-test-clean https://paperswithcode.com/sota/speech-recognition-on-libris... It seems like sota is 2.20% word error rate
- lunixbochs 7y agoI like reading through https://github.com/syhw/wer_are_we https://github.com/syhw/wer_are_we sometimes
- wstuartcl 7y agoNote that the leaders there are not "near real time" streaming type implementations like this is. They have the benefit of having full context (they act on the full input not a stream of current state). 7.5% is really good for this type of low latency stream based system.
- est31 7y agoLibriSpeech test clean is without background noise, with a good microphone, and most likely biased to native speakers (it is 50/50 male/female at least). In summary, it's not what an ASR engine will encounter. However, the dataset is used as a common method to benchmark ASR engines and thus that number is valuable. The SpecAugment paper by Google for example claims a 2.5% word error rate [1] on that corpus (using only LibriSpeech for training data). [1]: https://arxiv.org/pdf/1904.08779.pdf https://arxiv.org/pdf/1904.08779.pdf
- dabinat 7y agoIt’s worth mentioning that while DeepSpeech 0.6 supports SpecAugment, the released model was not trained with SpecAugment switched on.
- reubenmorais 7y agoAuthor here. I would add to what the sibling comments have mentioned by saying that SotA results should be taken with a grain of salt. Our engine is capable of streaming (processing the audio as it's being recorded), which is not doable with architectures that have bidirectional decoders or attention mechanisms that require the whole encoder input ahead of time. For real world applications, this is absolutely crucial, users want latency numbers on the order of milliseconds, not seconds. This is why, if you run a standard test set like LibriSpeech on, say, a commercial offering from Google, it will perform considerably worse than state of the art according to Google papers. This repository [0] has a benchmark of some commercial offerings. Our model beats all of those on Librispeech clean and other (except for Speechmatics on Librispeech clean), as well as on Common Voice. But note that the Common Voice corpus used in that benchmark is very old. In sum, I would compare this against solutions that go for the same space: fast, client-side ASR, rather than state of the art. [0] https://github.com/Franck-Dernoncourt/ASR_benchmark#benchmark-results https://github.com/Franck-Dernoncourt/ASR_benchmark#benchmar...
- albertzeyer 7y agoHi! I'm from the RWTH team (https://paperswithcode.com/sota/speech-recognition-on-librispeech-test-clean https://paperswithcode.com/sota/speech-recognition-on-libris...). Our best system (2.3% WER) (trained only on the Librispeech data, i.e. much less data than your model) currently is a hybrid HMM/NN model, and you are right, the acoustic model uses a BLSTM. However, we have shown in other work (https://www-i6.informatik.rwth-aachen.de/publications/download/1009/Zeyer--2016.pdf https://www-i6.informatik.rwth-aachen.de/publications/downlo...) that you can expect similar performance in an online system, or not much worse (there is also many related work, which often is maybe 5-10% relative worse than the offline system). I guess you prefer an "end-to-end" model over a hybrid HMM/NN model, for simplicity, right? As far as I remember, you use CTC, right? I always wondered why you have chosen CTC, and not some better model, like RNN-T, RNA, or some of the streaming attention variants. They should give you all the same properties (online capable, low latency, simple, end-to-end), but much better WER performance. Or is this simply because there currently is no simple ready-to-use implementation for those? Note that we published some TF code recently for some streaming attention variants, and plan to publish some RNN-T/RNA code soon.
- gok 7y ago7.5% WER is much much worse than the state of the art, but it's also important to recognize that LibriSpeech is kind of a ridiculous data set. It's 1000 hours of volunteers reading public domain novels out loud. It's very unlikely to match your use case. It's widely used in academia purely because it's larger than other freely available data sets.