3 ms·

The reason those kaldi results are noticeably worse is because they're not using RNN rescoring. The E2E models also take far more compute to train (facebook and

The reason those kaldi results are noticeably worse is because they're not using RNN rescoring. The E2E models also take far more compute to train (facebook and google train their models for hundreds of epochs, kaldi for... 4!). All the papers you read game results by adding unpruned ngram LMs or massive NNLMs. Point being if you put some more effort I believe you can get better results with kaldi. Remember we're discussing with which toolkit one can achieve better performance (given a certain amount of effort), which is not the same as just taking the results from different teams some of which have spent an order of magnitude more compute in gaming the results. For SOTA results with a hybrid approach check out RWTH's stuff. The kaldi people have recently been busy rewriting their backend to work with pytorch. In general using librispeech to gauge performance is a bad idea. It's read speech. I prefer to base my opinions on real world datasets. Have to say I am impressed by how good their (FB) results are without using an LM though.

lunixbochs 7y ago
I agree that librispeech isn’t the best benchmark, but it’s the benchmark we have. If you have a significantly better benchmark that both wav2letter SOTA and Kaldi SOTA have participated in, I’d love to see it. Without some kind of numbers cited, it’s easier to assume you’re speculating. Especially because, as I think we both acknowledge, wav2letter requires a lot of training power to actually use, so I doubt you’ve done the training yourself in the same way you’ve worked with Kaldi to be able to talk about it authoritatively. (One of my goals with releasing w2l models is that people will be able to do e.g. transfer learning or distillation and not repeat the expensive up front training themselves) RWTH has an impressive entry on the wer_are_we list for librispeech (2.3 / 5.0), but unfortunately I couldn’t find a way to actually use that research without a lot of work. Facebook has easy recipes posted that reproduce their results, which have been quite nice to work with. RWTH, Google, and Capio held an impressive top spot in the results for a while, which might be where you’re basing your judgement. I don’t think the biggest benefit of wav2letter was the architecture (until their SOTA paper), I think it was the speed and simplicity. I managed to port the original wav2letter inference to numpy/pytorch in two small files (same model weights, same inference output) because it was such a simple convolutional architecture. (The only fiddly part was matching the output of their featurization code) I agree the data and training overhead is a big aspect, which is why I’ve been training and releasing freely available wav2letter models on far more data than what Facebook has published using librispeech. My upcoming goal is to reproduce the SOTA results but with stronger training data. My baseline data is around 4000h currently and is far more diverse than librispeech. Unfortunately I’m independent and can’t afford the LDC datasets, though I have some neat ideas for extending FB’s semi supervised work to even more data.
- ftreml 7y ago
  are there any wav2letter models ready for download ?
  - lunixbochs 7y ago
    Facebook posted some of the models listed in their SOTA paper already. I haven’t tried these yet. https://github.com/facebookresearch/wav2letter/blob/master/recipes/models/sota/2019/README.md https://github.com/facebookresearch/wav2letter/blob/master/r... My models are here: https://talonvoice.com/research/ https://talonvoice.com/research/ I haven’t yet posted the model I’ve been working on most recently. I’m 120 epochs into a large size model trained on all of my datasets. I also have another 1000-1500h of audio I haven’t finished prepping to train on. Here is a web demo. It’s currently running a slightly older checkpoint of my WIP large model, and the deepspeech LM: https://web2letter-west-1.talonvoice.com/ https://web2letter-west-1.talonvoice.com/
    - ftreml 7y ago
      so this is a real cool project. as soon as you finished your training I will be happy to add it as option (or maybe default setup) in the botium speech processing setup (if you want that). Do you have any experience with online decoding in wav2letter ? Is there something like a Websocket API available somewhere ?
      - lunixbochs 7y ago
        How does accuracy compare to what you’re using now?
        ftreml 7y ago
        the german model is from the kaldi tuda recipe with WER of 15%. the english is from the tedlium recipe with WER of 7%. room for improvement, but for our original purpose it was sufficient.
- 7y ago

lunixbochs 7y ago

I agree that librispeech isn’t the best benchmark, but it’s the benchmark we have. If you have a significantly better benchmark that both wav2letter SOTA and Kaldi SOTA have participated in, I’d love to see it. Without some kind of numbers cited, it’s easier to assume you’re speculating. Especially because, as I think we both acknowledge, wav2letter requires a lot of training power to actually use, so I doubt you’ve done the training yourself in the same way you’ve worked with Kaldi to be able to talk about it authoritatively. (One of my goals with releasing w2l models is that people will be able to do e.g. transfer learning or distillation and not repeat the expensive up front training themselves) RWTH has an impressive entry on the wer_are_we list for librispeech (2.3 / 5.0), but unfortunately I couldn’t find a way to actually use that research without a lot of work. Facebook has easy recipes posted that reproduce their results, which have been quite nice to work with. RWTH, Google, and Capio held an impressive top spot in the results for a while, which might be where you’re basing your judgement. I don’t think the biggest benefit of wav2letter was the architecture (until their SOTA paper), I think it was the speed and simplicity. I managed to port the original wav2letter inference to numpy/pytorch in two small files (same model weights, same inference output) because it was such a simple convolutional architecture. (The only fiddly part was matching the output of their featurization code) I agree the data and training overhead is a big aspect, which is why I’ve been training and releasing freely available wav2letter models on far more data than what Facebook has published using librispeech. My upcoming goal is to reproduce the SOTA results but with stronger training data. My baseline data is around 4000h currently and is far more diverse than librispeech. Unfortunately I’m independent and can’t afford the LDC datasets, though I have some neat ideas for extending FB’s semi supervised work to even more data.

ftreml 7y ago
are there any wav2letter models ready for download ?
- lunixbochs 7y ago
  Facebook posted some of the models listed in their SOTA paper already. I haven’t tried these yet. https://github.com/facebookresearch/wav2letter/blob/master/recipes/models/sota/2019/README.md https://github.com/facebookresearch/wav2letter/blob/master/r... My models are here: https://talonvoice.com/research/ https://talonvoice.com/research/ I haven’t yet posted the model I’ve been working on most recently. I’m 120 epochs into a large size model trained on all of my datasets. I also have another 1000-1500h of audio I haven’t finished prepping to train on. Here is a web demo. It’s currently running a slightly older checkpoint of my WIP large model, and the deepspeech LM: https://web2letter-west-1.talonvoice.com/ https://web2letter-west-1.talonvoice.com/
  - ftreml 7y ago
    so this is a real cool project. as soon as you finished your training I will be happy to add it as option (or maybe default setup) in the botium speech processing setup (if you want that). Do you have any experience with online decoding in wav2letter ? Is there something like a Websocket API available somewhere ?
    - lunixbochs 7y ago
      How does accuracy compare to what you’re using now?
      - ftreml 7y ago
        the german model is from the kaldi tuda recipe with WER of 15%. the english is from the tedlium recipe with WER of 7%. room for improvement, but for our original purpose it was sufficient.
7y ago