6 ms·

Facebook has released wav2letter++. I'd wager that will outperform kaldi by a wide margin.

ftreml 7y ago
in our tests the performance was comparable, so we had no reason to switch from kaldi to something else (german data only). after all, what matters the most is the available training data. are there ready trained models available for wav2letter ?
- grizzles 7y ago
  Oops, there I go being Anglocentric again. The datasets are out there; but idk if there are German trained models. Fortunately wav2letter++ looks like it's been built to be very friendly to train a more extensive one if there isn't. My understanding is they are still releasing parts of it.
dsteinman 7y ago
Any idea how it compares to Mozilla's DeepSpeech?
- ftreml 7y ago
  from my experience, when trained with the same data, kaldi is slightly better and with custom recipes adaptable to changing conditions. deepspeech has way better documentation and is more developer friendly. wav2letter seems to be the quickest. i guess there is no real winner here ... depends what criteria are applied. just an example, kaldi is a weird mixture of c++, python2, python3, shell scripts, java, perl. hard to oversee. deepspeech is python. wav2letter is an exe file.
  - magicalhippo 7y ago
    > deepspeech is python I was under the impression DeepSpeech was native (C++), with bindings for Python and others. Personally I've used it with Node.js so far, and I couldn't see any dependencies on Python. edit: I was talking client, you're talking training I guess.
Nimitz14 7y ago
It is very unlikely a E2E toolkit can outperform kaldi if they're trained on the same data. Also I suspect these guys aren't using the latest kaldi architectures.
- ftreml 7y ago
  we used existing recipies for training (tuda german), slightly adapted and with additional training data. i am sure with more knowledge we would have made some extra percent ... performance is not really good, but good enough for our purposes, and with a satisfying cost vs quality trade off
  - Nimitz14 7y ago
    You should look at the latest recipe's for tedlium, librispeech, swbd etc you'll get significantly better results with those models.
- grizzles 7y ago
  What's your reasoning? Google's latest published research is analogously parsimonious with this approach. They are doing decoding using a simple beam search through a single neural network. See https://ai.googleblog.com/2019/03/an-all-neural-on-device-speech.html https://ai.googleblog.com/2019/03/an-all-neural-on-device-sp...
  - ftreml 7y ago
    very very interesting, didnt know this one. as soon as i am having some hours left will try to run some evaluation on this. after all, for my project only performance in german language counts
- lunixbochs 7y ago
  Did you miss wav2letter achieving a new 2019 SOTA on librispeech? Both projects have a _lot_ of architectures and papers in play, I know it can be hard to keep up. Aren’t some of the peak Kaldi numbers also with _huge_ language models that aren’t so practical to deploy? The best kaldi results I’ve found here both use “fglarge” LMs, but a quick search didn’t tell me how big that actually is. The best I can find for the test set are (from main RESULTS): test-clean: 4.31% WER test-other: 10.62% WER From here (not sure why buried): https://github.com/kaldi-asr/kaldi/blob/master/egs/librispeech/s5/local/chain/tuning/run_tdnn_1d.sh https://github.com/kaldi-asr/kaldi/blob/master/egs/librispee... test-clean: 3.80% WER test-other: 8.76% WER This is the 2019 wav2letter SOTA: https://github.com/facebookresearch/wav2letter/blob/master/recipes/models/sota/2019/README.md https://github.com/facebookresearch/wav2letter/blob/master/r... Sure, one of their techniques is to semi supervised train on 60k hours of audio, so I’ll post their best numbers both with and without that step (as you said “if kaldi is trained on the same data”) Without 60k unsupervised audio: test-clean: 2.31% WER test-other: 5.18% WER With 60k audio: test-clean: 2.03% WER test-other: 4.11% WER That’s so far ahead of kaldi’s librispeech RESULTS, let’s go up the list and look at their acoustic model tests that don’t even bother with a language model: Without 60k or language model: (test-clean / test-other WER) 3.05 / 7.01 With 60k without language model: 2.30 / 5.29 Am I missing a kaldi whitepaper here? I found “State-of-the-Art Speech Recognition Using Multi-Stream Self-Attention With Dilated 1D Convolutions” on wer_are_we claiming 2.2 / 5.8 using Kaldi as a base, which is impressive, but they changed the network architecture and I didn’t find this in any of the open-source kaldi librispeech subdirectories. (If arbitrary model changes count, at some point you might as well treat both wav2letter and kaldi as “tensorflow with some speech helpers” because I’m sure you can implement either framework’s best neural architecture in the other framework.) If there’s a better paper on kaldi than 2.2 / 5.8 you should comment with it and PR it to wer_are_we, because wav2letter is currently beating every other LS result posted there.

in our tests the performance was comparable, so we had no reason to switch from kaldi to something else (german data only). after all, what matters the most is the available training data. are there ready trained models available for wav2letter ?

grizzles 7y ago
Oops, there I go being Anglocentric again. The datasets are out there; but idk if there are German trained models. Fortunately wav2letter++ looks like it's been built to be very friendly to train a more extensive one if there isn't. My understanding is they are still releasing parts of it.

dsteinman 7y ago

Any idea how it compares to Mozilla's DeepSpeech?

ftreml 7y ago
from my experience, when trained with the same data, kaldi is slightly better and with custom recipes adaptable to changing conditions. deepspeech has way better documentation and is more developer friendly. wav2letter seems to be the quickest. i guess there is no real winner here ... depends what criteria are applied. just an example, kaldi is a weird mixture of c++, python2, python3, shell scripts, java, perl. hard to oversee. deepspeech is python. wav2letter is an exe file.
- magicalhippo 7y ago
  > deepspeech is python I was under the impression DeepSpeech was native (C++), with bindings for Python and others. Personally I've used it with Node.js so far, and I couldn't see any dependencies on Python. edit: I was talking client, you're talking training I guess.

Nimitz14 7y ago

It is very unlikely a E2E toolkit can outperform kaldi if they're trained on the same data. Also I suspect these guys aren't using the latest kaldi architectures.

ftreml 7y ago
we used existing recipies for training (tuda german), slightly adapted and with additional training data. i am sure with more knowledge we would have made some extra percent ... performance is not really good, but good enough for our purposes, and with a satisfying cost vs quality trade off
- Nimitz14 7y ago
  You should look at the latest recipe's for tedlium, librispeech, swbd etc you'll get significantly better results with those models.
grizzles 7y ago
What's your reasoning? Google's latest published research is analogously parsimonious with this approach. They are doing decoding using a simple beam search through a single neural network. See https://ai.googleblog.com/2019/03/an-all-neural-on-device-speech.html https://ai.googleblog.com/2019/03/an-all-neural-on-device-sp...
- ftreml 7y ago
  very very interesting, didnt know this one. as soon as i am having some hours left will try to run some evaluation on this. after all, for my project only performance in german language counts
lunixbochs 7y ago
Did you miss wav2letter achieving a new 2019 SOTA on librispeech? Both projects have a _lot_ of architectures and papers in play, I know it can be hard to keep up. Aren’t some of the peak Kaldi numbers also with _huge_ language models that aren’t so practical to deploy? The best kaldi results I’ve found here both use “fglarge” LMs, but a quick search didn’t tell me how big that actually is. The best I can find for the test set are (from main RESULTS): test-clean: 4.31% WER test-other: 10.62% WER From here (not sure why buried): https://github.com/kaldi-asr/kaldi/blob/master/egs/librispeech/s5/local/chain/tuning/run_tdnn_1d.sh https://github.com/kaldi-asr/kaldi/blob/master/egs/librispee... test-clean: 3.80% WER test-other: 8.76% WER This is the 2019 wav2letter SOTA: https://github.com/facebookresearch/wav2letter/blob/master/recipes/models/sota/2019/README.md https://github.com/facebookresearch/wav2letter/blob/master/r... Sure, one of their techniques is to semi supervised train on 60k hours of audio, so I’ll post their best numbers both with and without that step (as you said “if kaldi is trained on the same data”) Without 60k unsupervised audio: test-clean: 2.31% WER test-other: 5.18% WER With 60k audio: test-clean: 2.03% WER test-other: 4.11% WER That’s so far ahead of kaldi’s librispeech RESULTS, let’s go up the list and look at their acoustic model tests that don’t even bother with a language model: Without 60k or language model: (test-clean / test-other WER) 3.05 / 7.01 With 60k without language model: 2.30 / 5.29 Am I missing a kaldi whitepaper here? I found “State-of-the-Art Speech Recognition Using Multi-Stream Self-Attention With Dilated 1D Convolutions” on wer_are_we claiming 2.2 / 5.8 using Kaldi as a base, which is impressive, but they changed the network architecture and I didn’t find this in any of the open-source kaldi librispeech subdirectories. (If arbitrary model changes count, at some point you might as well treat both wav2letter and kaldi as “tensorflow with some speech helpers” because I’m sure you can implement either framework’s best neural architecture in the other framework.) If there’s a better paper on kaldi than 2.2 / 5.8 you should comment with it and PR it to wer_are_we, because wav2letter is currently beating every other LS result posted there.