3 ms·

As another practitioner, I find it odd that the author mentions frameworks such as OpenNMT but not Kaldi when comparing SST toolkits/frameworks. Overall, I get

As another practitioner, I find it odd that the author mentions frameworks such as OpenNMT but not Kaldi when comparing SST toolkits/frameworks. Overall, I get the feeling that the author hasn't worked with speech data for too long. I agree with some of the general points the author makes, such as papers having tons of equations just for equations sake, reproducibility, not enough details to reproduce (in a conference paper..), chasing SOTA on some dataset, but these aren't limited to speech processing research. Large companies don't release their internal datasets for voice search (Google Assistant, Alexa, etc.), call center, or clinical applications, as there is no incentive to do so and also because they likely can't for licensing and data rights issues. By the way, what's the situation for the author's OpenSTT corpus? Did the people speaking on the thousands of hours of phone calls release them under a free license?

Nimitz14 7y ago
Yup. Copying my post from reddit: Seems like the title should really be criticisms of STT groups in the industry, work from academic groups is ignored. The original article was kind of silly anyways, it was all engineering work, very little research. Engineering work is important, but you're not going to get an image net moment in ASR by twiddling with hyperparameters or reducing training time. Also a bizarre decision to pick deepspeech as a framework, of course your models will take forever to train and won't even be that good in the end. It's end-to-end and hasn't been SOTA for a while. > It is hard to include the original Deep Speech paper in this list, mainly because of how many different things they tried, popularized, and pioneered. Had to laugh at this. What exactly did they "popularize and pioneer" other than the practice of training and reporting results on very large private datasets? [This, among others, was a much more important end-to-end paper I think.](https://arxiv.org/pdf/1303.5778.pdf https://arxiv.org/pdf/1303.5778.pdf) There are some good points about using private datasets and overfitting on a read-speech dataset, and using very large models etc. but I really wish they would have used their data to train a kaldi system. I can guess why they didn't, they have no background in speech and found it too hard to use. Still disappointing. Anyways, I would argue the reason that a "imagenet moment" hasn't arrived for ASR is that vision is universal, but speech splits into different languages, making it much harder to build a single model that everyone else can use as a seed model. I believe multilingual models are the future.
qayxc 7y ago
> As another practitioner, I find it odd that the author mentions frameworks such as OpenNMT but not Kaldi when comparing SST toolkits/frameworks. [unpopular opinion ahead] Maybe that's because Kaldi represents the ultimate culmination of the author's critique: • the provided scripts are useless to anyone without access to the academic datasets they use • Kaldi is very cumbersome and overly complex to use and adapt (most of the toolchain relies on exact copies of entire directory trees) • Kaldi is a research tool by researchers for researchers and not in any way shape or form aimed at practitioners in search of a deployable solution • Kaldi is very poorly documented (from a user perspective) and focuses on recipes for your own experiments, not writing applications and roll out In summary, Kaldi isn't a practical toolkit. It's a framework for R&D on speech-to-text.
- nshm 7y ago
  > the provided scripts are useless to anyone without access to the academic datasets they use Kaldi team shares all datasets when possible on http://openslr.org http://openslr.org, a major collection of speech datasets. Librispeech and Tedlium were major breakthrough in their times. When everyone in research trained on 80 hours WSJ and Google trained on 2000 hours private dataset, 1000 hours of librispeech was a game changer. > Kaldi is very cumbersome and overly complex to use and adapt (most of the toolchain relies on exact copies of entire directory trees) On the other hand the Kaldi API is very stable. You still can run 4 year old recipes with simple run.sh. Any tensorflow software gets obsolete once in 3 months when TF API changes. Kaldi recipe results are usually reproducible up to the digit and you can even see the tuning history (which features did improve, which didn't). > Kaldi is a research tool by researchers for researchers and not in any way shape or form aimed at practitioners in search of a deployable solution There are hundreds of companies all over the world building practical solutions with Kaldi. The only thing that Kaldi team should do is to ask users to mention it. > Kaldi is very poorly documented (from a user perspective) and focuses on recipes for your own experiments, not writing applications and roll out There are also dozen of projects on Github which enable use of Kaldi with simple pip install or docker pull. kaldi-gstreamer-server, kaldi-active-grammar, vosk and many others.
- albertzeyer 7y ago
  (Disclaimer: I also work in the field.) I would disagree with most of your points. The provided scripts are available for a huge number of datasets. Many of the datasets are also useful for use in production. And it's very simple to adapt the provided scripts for further data. Kaldi is complex and big, yes. But you don't have to use all of it. And it is that way because doing STT properly simply is complex. At least for the conventional standard hybrid HMM/NN models. You could surely go some of the end-to-end ways but that will lead to other problems in production. And once you need to deal with these problems, it might become even more complicated. (This is a whole topic on its own...) Kaldi is used in research as well as in production. It can be easily deployed. Kaldi is very well documented. Just check the homepage of it. In addition to it, there is a huge community (both academia and industry).

Nimitz14 7y ago

Yup. Copying my post from reddit: Seems like the title should really be criticisms of STT groups in the industry, work from academic groups is ignored. The original article was kind of silly anyways, it was all engineering work, very little research. Engineering work is important, but you're not going to get an image net moment in ASR by twiddling with hyperparameters or reducing training time. Also a bizarre decision to pick deepspeech as a framework, of course your models will take forever to train and won't even be that good in the end. It's end-to-end and hasn't been SOTA for a while. > It is hard to include the original Deep Speech paper in this list, mainly because of how many different things they tried, popularized, and pioneered. Had to laugh at this. What exactly did they "popularize and pioneer" other than the practice of training and reporting results on very large private datasets? [This, among others, was a much more important end-to-end paper I think.](https://arxiv.org/pdf/1303.5778.pdf https://arxiv.org/pdf/1303.5778.pdf) There are some good points about using private datasets and overfitting on a read-speech dataset, and using very large models etc. but I really wish they would have used their data to train a kaldi system. I can guess why they didn't, they have no background in speech and found it too hard to use. Still disappointing. Anyways, I would argue the reason that a "imagenet moment" hasn't arrived for ASR is that vision is universal, but speech splits into different languages, making it much harder to build a single model that everyone else can use as a seed model. I believe multilingual models are the future.

qayxc 7y ago

> As another practitioner, I find it odd that the author mentions frameworks such as OpenNMT but not Kaldi when comparing SST toolkits/frameworks. [unpopular opinion ahead] Maybe that's because Kaldi represents the ultimate culmination of the author's critique: • the provided scripts are useless to anyone without access to the academic datasets they use • Kaldi is very cumbersome and overly complex to use and adapt (most of the toolchain relies on exact copies of entire directory trees) • Kaldi is a research tool by researchers for researchers and not in any way shape or form aimed at practitioners in search of a deployable solution • Kaldi is very poorly documented (from a user perspective) and focuses on recipes for your own experiments, not writing applications and roll out In summary, Kaldi isn't a practical toolkit. It's a framework for R&D on speech-to-text.

nshm 7y ago
> the provided scripts are useless to anyone without access to the academic datasets they use Kaldi team shares all datasets when possible on http://openslr.org http://openslr.org, a major collection of speech datasets. Librispeech and Tedlium were major breakthrough in their times. When everyone in research trained on 80 hours WSJ and Google trained on 2000 hours private dataset, 1000 hours of librispeech was a game changer. > Kaldi is very cumbersome and overly complex to use and adapt (most of the toolchain relies on exact copies of entire directory trees) On the other hand the Kaldi API is very stable. You still can run 4 year old recipes with simple run.sh. Any tensorflow software gets obsolete once in 3 months when TF API changes. Kaldi recipe results are usually reproducible up to the digit and you can even see the tuning history (which features did improve, which didn't). > Kaldi is a research tool by researchers for researchers and not in any way shape or form aimed at practitioners in search of a deployable solution There are hundreds of companies all over the world building practical solutions with Kaldi. The only thing that Kaldi team should do is to ask users to mention it. > Kaldi is very poorly documented (from a user perspective) and focuses on recipes for your own experiments, not writing applications and roll out There are also dozen of projects on Github which enable use of Kaldi with simple pip install or docker pull. kaldi-gstreamer-server, kaldi-active-grammar, vosk and many others.
albertzeyer 7y ago
(Disclaimer: I also work in the field.) I would disagree with most of your points. The provided scripts are available for a huge number of datasets. Many of the datasets are also useful for use in production. And it's very simple to adapt the provided scripts for further data. Kaldi is complex and big, yes. But you don't have to use all of it. And it is that way because doing STT properly simply is complex. At least for the conventional standard hybrid HMM/NN models. You could surely go some of the end-to-end ways but that will lead to other problems in production. And once you need to deal with these problems, it might become even more complicated. (This is a whole topic on its own...) Kaldi is used in research as well as in production. It can be easily deployed. Kaldi is very well documented. Just check the homepage of it. In addition to it, there is a huge community (both academia and industry).