6 ms·

It is very unlikely a E2E toolkit can outperform kaldi if they're trained on the same data. Also I suspect these guys aren't using the latest kaldi architectur

It is very unlikely a E2E toolkit can outperform kaldi if they're trained on the same data. Also I suspect these guys aren't using the latest kaldi architectures.

ftreml 7y ago
we used existing recipies for training (tuda german), slightly adapted and with additional training data. i am sure with more knowledge we would have made some extra percent ... performance is not really good, but good enough for our purposes, and with a satisfying cost vs quality trade off
- Nimitz14 7y ago
  You should look at the latest recipe's for tedlium, librispeech, swbd etc you'll get significantly better results with those models.
grizzles 7y ago
What's your reasoning? Google's latest published research is analogously parsimonious with this approach. They are doing decoding using a simple beam search through a single neural network. See https://ai.googleblog.com/2019/03/an-all-neural-on-device-speech.html https://ai.googleblog.com/2019/03/an-all-neural-on-device-sp...
- ftreml 7y ago
  very very interesting, didnt know this one. as soon as i am having some hours left will try to run some evaluation on this. after all, for my project only performance in german language counts
lunixbochs 7y ago
Did you miss wav2letter achieving a new 2019 SOTA on librispeech? Both projects have a _lot_ of architectures and papers in play, I know it can be hard to keep up. Aren’t some of the peak Kaldi numbers also with _huge_ language models that aren’t so practical to deploy? The best kaldi results I’ve found here both use “fglarge” LMs, but a quick search didn’t tell me how big that actually is. The best I can find for the test set are (from main RESULTS): test-clean: 4.31% WER test-other: 10.62% WER From here (not sure why buried): https://github.com/kaldi-asr/kaldi/blob/master/egs/librispeech/s5/local/chain/tuning/run_tdnn_1d.sh https://github.com/kaldi-asr/kaldi/blob/master/egs/librispee... test-clean: 3.80% WER test-other: 8.76% WER This is the 2019 wav2letter SOTA: https://github.com/facebookresearch/wav2letter/blob/master/recipes/models/sota/2019/README.md https://github.com/facebookresearch/wav2letter/blob/master/r... Sure, one of their techniques is to semi supervised train on 60k hours of audio, so I’ll post their best numbers both with and without that step (as you said “if kaldi is trained on the same data”) Without 60k unsupervised audio: test-clean: 2.31% WER test-other: 5.18% WER With 60k audio: test-clean: 2.03% WER test-other: 4.11% WER That’s so far ahead of kaldi’s librispeech RESULTS, let’s go up the list and look at their acoustic model tests that don’t even bother with a language model: Without 60k or language model: (test-clean / test-other WER) 3.05 / 7.01 With 60k without language model: 2.30 / 5.29 Am I missing a kaldi whitepaper here? I found “State-of-the-Art Speech Recognition Using Multi-Stream Self-Attention With Dilated 1D Convolutions” on wer_are_we claiming 2.2 / 5.8 using Kaldi as a base, which is impressive, but they changed the network architecture and I didn’t find this in any of the open-source kaldi librispeech subdirectories. (If arbitrary model changes count, at some point you might as well treat both wav2letter and kaldi as “tensorflow with some speech helpers” because I’m sure you can implement either framework’s best neural architecture in the other framework.) If there’s a better paper on kaldi than 2.2 / 5.8 you should comment with it and PR it to wer_are_we, because wav2letter is currently beating every other LS result posted there.
- Nimitz14 7y ago
  The reason those kaldi results are noticeably worse is because they're not using RNN rescoring. The E2E models also take far more compute to train (facebook and google train their models for hundreds of epochs, kaldi for... 4!). All the papers you read game results by adding unpruned ngram LMs or massive NNLMs. Point being if you put some more effort I believe you can get better results with kaldi. Remember we're discussing with which toolkit one can achieve better performance (given a certain amount of effort), which is not the same as just taking the results from different teams some of which have spent an order of magnitude more compute in gaming the results. For SOTA results with a hybrid approach check out RWTH's stuff. The kaldi people have recently been busy rewriting their backend to work with pytorch. In general using librispeech to gauge performance is a bad idea. It's read speech. I prefer to base my opinions on real world datasets. Have to say I am impressed by how good their (FB) results are without using an LM though.
  - lunixbochs 7y ago
    I agree that librispeech isn’t the best benchmark, but it’s the benchmark we have. If you have a significantly better benchmark that both wav2letter SOTA and Kaldi SOTA have participated in, I’d love to see it. Without some kind of numbers cited, it’s easier to assume you’re speculating. Especially because, as I think we both acknowledge, wav2letter requires a lot of training power to actually use, so I doubt you’ve done the training yourself in the same way you’ve worked with Kaldi to be able to talk about it authoritatively. (One of my goals with releasing w2l models is that people will be able to do e.g. transfer learning or distillation and not repeat the expensive up front training themselves) RWTH has an impressive entry on the wer_are_we list for librispeech (2.3 / 5.0), but unfortunately I couldn’t find a way to actually use that research without a lot of work. Facebook has easy recipes posted that reproduce their results, which have been quite nice to work with. RWTH, Google, and Capio held an impressive top spot in the results for a while, which might be where you’re basing your judgement. I don’t think the biggest benefit of wav2letter was the architecture (until their SOTA paper), I think it was the speed and simplicity. I managed to port the original wav2letter inference to numpy/pytorch in two small files (same model weights, same inference output) because it was such a simple convolutional architecture. (The only fiddly part was matching the output of their featurization code) I agree the data and training overhead is a big aspect, which is why I’ve been training and releasing freely available wav2letter models on far more data than what Facebook has published using librispeech. My upcoming goal is to reproduce the SOTA results but with stronger training data. My baseline data is around 4000h currently and is far more diverse than librispeech. Unfortunately I’m independent and can’t afford the LDC datasets, though I have some neat ideas for extending FB’s semi supervised work to even more data.

It is very unlikely a E2E toolkit can outperform kaldi if they're trained on the same data. Also I suspect these guys aren't using the latest kaldi architectures.

ftreml 7y ago

we used existing recipies for training (tuda german), slightly adapted and with additional training data. i am sure with more knowledge we would have made some extra percent ... performance is not really good, but good enough for our purposes, and with a satisfying cost vs quality trade off

Nimitz14 7y ago
You should look at the latest recipe's for tedlium, librispeech, swbd etc you'll get significantly better results with those models.

grizzles 7y ago

What's your reasoning? Google's latest published research is analogously parsimonious with this approach. They are doing decoding using a simple beam search through a single neural network. See https://ai.googleblog.com/2019/03/an-all-neural-on-device-speech.html https://ai.googleblog.com/2019/03/an-all-neural-on-device-sp...

ftreml 7y ago
very very interesting, didnt know this one. as soon as i am having some hours left will try to run some evaluation on this. after all, for my project only performance in german language counts

lunixbochs 7y ago

Did you miss wav2letter achieving a new 2019 SOTA on librispeech? Both projects have a _lot_ of architectures and papers in play, I know it can be hard to keep up. Aren’t some of the peak Kaldi numbers also with _huge_ language models that aren’t so practical to deploy? The best kaldi results I’ve found here both use “fglarge” LMs, but a quick search didn’t tell me how big that actually is. The best I can find for the test set are (from main RESULTS): test-clean: 4.31% WER test-other: 10.62% WER From here (not sure why buried): https://github.com/kaldi-asr/kaldi/blob/master/egs/librispeech/s5/local/chain/tuning/run_tdnn_1d.sh https://github.com/kaldi-asr/kaldi/blob/master/egs/librispee... test-clean: 3.80% WER test-other: 8.76% WER This is the 2019 wav2letter SOTA: https://github.com/facebookresearch/wav2letter/blob/master/recipes/models/sota/2019/README.md https://github.com/facebookresearch/wav2letter/blob/master/r... Sure, one of their techniques is to semi supervised train on 60k hours of audio, so I’ll post their best numbers both with and without that step (as you said “if kaldi is trained on the same data”) Without 60k unsupervised audio: test-clean: 2.31% WER test-other: 5.18% WER With 60k audio: test-clean: 2.03% WER test-other: 4.11% WER That’s so far ahead of kaldi’s librispeech RESULTS, let’s go up the list and look at their acoustic model tests that don’t even bother with a language model: Without 60k or language model: (test-clean / test-other WER) 3.05 / 7.01 With 60k without language model: 2.30 / 5.29 Am I missing a kaldi whitepaper here? I found “State-of-the-Art Speech Recognition Using Multi-Stream Self-Attention With Dilated 1D Convolutions” on wer_are_we claiming 2.2 / 5.8 using Kaldi as a base, which is impressive, but they changed the network architecture and I didn’t find this in any of the open-source kaldi librispeech subdirectories. (If arbitrary model changes count, at some point you might as well treat both wav2letter and kaldi as “tensorflow with some speech helpers” because I’m sure you can implement either framework’s best neural architecture in the other framework.) If there’s a better paper on kaldi than 2.2 / 5.8 you should comment with it and PR it to wer_are_we, because wav2letter is currently beating every other LS result posted there.

Nimitz14 7y ago
The reason those kaldi results are noticeably worse is because they're not using RNN rescoring. The E2E models also take far more compute to train (facebook and google train their models for hundreds of epochs, kaldi for... 4!). All the papers you read game results by adding unpruned ngram LMs or massive NNLMs. Point being if you put some more effort I believe you can get better results with kaldi. Remember we're discussing with which toolkit one can achieve better performance (given a certain amount of effort), which is not the same as just taking the results from different teams some of which have spent an order of magnitude more compute in gaming the results. For SOTA results with a hybrid approach check out RWTH's stuff. The kaldi people have recently been busy rewriting their backend to work with pytorch. In general using librispeech to gauge performance is a bad idea. It's read speech. I prefer to base my opinions on real world datasets. Have to say I am impressed by how good their (FB) results are without using an LM though.
- lunixbochs 7y ago
  I agree that librispeech isn’t the best benchmark, but it’s the benchmark we have. If you have a significantly better benchmark that both wav2letter SOTA and Kaldi SOTA have participated in, I’d love to see it. Without some kind of numbers cited, it’s easier to assume you’re speculating. Especially because, as I think we both acknowledge, wav2letter requires a lot of training power to actually use, so I doubt you’ve done the training yourself in the same way you’ve worked with Kaldi to be able to talk about it authoritatively. (One of my goals with releasing w2l models is that people will be able to do e.g. transfer learning or distillation and not repeat the expensive up front training themselves) RWTH has an impressive entry on the wer_are_we list for librispeech (2.3 / 5.0), but unfortunately I couldn’t find a way to actually use that research without a lot of work. Facebook has easy recipes posted that reproduce their results, which have been quite nice to work with. RWTH, Google, and Capio held an impressive top spot in the results for a while, which might be where you’re basing your judgement. I don’t think the biggest benefit of wav2letter was the architecture (until their SOTA paper), I think it was the speed and simplicity. I managed to port the original wav2letter inference to numpy/pytorch in two small files (same model weights, same inference output) because it was such a simple convolutional architecture. (The only fiddly part was matching the output of their featurization code) I agree the data and training overhead is a big aspect, which is why I’ve been training and releasing freely available wav2letter models on far more data than what Facebook has published using librispeech. My upcoming goal is to reproduce the SOTA results but with stronger training data. My baseline data is around 4000h currently and is far more diverse than librispeech. Unfortunately I’m independent and can’t afford the LDC datasets, though I have some neat ideas for extending FB’s semi supervised work to even more data.