6 ms·

Is it an issue of open source software being inadequate, or is it the lack of sufficient training data and compute power local to your home? I’m under the impr

by dsacco 9y ago

Is it an issue of open source software being inadequate, or is it the lack of sufficient training data and compute power local to your home? I’m under the impression that Google is mostly dogfooding its open source tooling for machine learning in GCP, and actually differentiates based on trained models and compute power.

backpropaganda 9y ago
A bit of both. When data doesn't exist, people aren't motivated enough to create the open source tools which would leverage it. While Librispeech is okay for academic research, it's not enough to create a good production-grade speech recognition system.
adrianbg 9y ago
The problem wrt open source / free solutions is data. Kaldi is open source and gets state of the art results -- but the data costs a lot of money. Training the models is doable on a commodity GPU although it takes quite a while.
- jjwiseman 9y ago
  I didn't realize Kaldi could get state of the art results. Do you say that because you know of people doing that, or is your comment based on knowing the architecture of Kaldi?
  - adrianbg 9y ago
    The 8.5% in this file is what you'd compare to Microsoft and IBM's recent ~5% results. https://github.com/kaldi-asr/kaldi/blob/master/egs/fisher_swbd/s5/RESULTS#L103 https://github.com/kaldi-asr/kaldi/blob/master/egs/fisher_sw... Kaldi hasn't been in first place on that dataset recently, but it was a few years ago. On other more researchy datasets (eg. for distant speakers or languages other than English), the best system is often based on Kaldi.
    - woodson 9y ago
      One reason for the discrepancy between quoted numbers is that, if you are only after pushing that number down and not particularly interested in getting a scalable system, then you are free to run as many systems as you like in as many configurations as possible and then try to combine their outputs (ROVER etc.).
      - adrianbg 9y ago
        Yeah. I don't get the impression that the Kaldi core team has been trying very hard recently to get SOTA on eval2k/switchboard. This number uses one acoustic model with a trigram LM decode + fourgram rescoring -- there isn't even a neural net language model in there. If I remember correctly, Microsoft's first "human parity" result used something like three acoustic models and at least four types of language models. This Kaldi model is competitive with the best single acoustic model Microsoft used.
        woodson 9y ago
        Fully agree. I think their work on training data augmentation (e.g., their ICASSP paper, http://danielpovey.com/files/2017_icassp_reverberation.pdf http://danielpovey.com/files/2017_icassp_reverberation.pdf, or the ASPiRE model before) has a bigger impact on the practical usefulness of ASR than getting an X% relative improvement over the previous SOTA on the eval2000 set.

by dsacco 9y ago

backpropaganda 9y ago

A bit of both. When data doesn't exist, people aren't motivated enough to create the open source tools which would leverage it. While Librispeech is okay for academic research, it's not enough to create a good production-grade speech recognition system.

adrianbg 9y ago

The problem wrt open source / free solutions is data. Kaldi is open source and gets state of the art results -- but the data costs a lot of money. Training the models is doable on a commodity GPU although it takes quite a while.

jjwiseman 9y ago
I didn't realize Kaldi could get state of the art results. Do you say that because you know of people doing that, or is your comment based on knowing the architecture of Kaldi?
- adrianbg 9y ago
  The 8.5% in this file is what you'd compare to Microsoft and IBM's recent ~5% results. https://github.com/kaldi-asr/kaldi/blob/master/egs/fisher_swbd/s5/RESULTS#L103 https://github.com/kaldi-asr/kaldi/blob/master/egs/fisher_sw... Kaldi hasn't been in first place on that dataset recently, but it was a few years ago. On other more researchy datasets (eg. for distant speakers or languages other than English), the best system is often based on Kaldi.
  - woodson 9y ago
    One reason for the discrepancy between quoted numbers is that, if you are only after pushing that number down and not particularly interested in getting a scalable system, then you are free to run as many systems as you like in as many configurations as possible and then try to combine their outputs (ROVER etc.).
    - adrianbg 9y ago
      Yeah. I don't get the impression that the Kaldi core team has been trying very hard recently to get SOTA on eval2k/switchboard. This number uses one acoustic model with a trigram LM decode + fourgram rescoring -- there isn't even a neural net language model in there. If I remember correctly, Microsoft's first "human parity" result used something like three acoustic models and at least four types of language models. This Kaldi model is competitive with the best single acoustic model Microsoft used.
      - woodson 9y ago
        Fully agree. I think their work on training data augmentation (e.g., their ICASSP paper, http://danielpovey.com/files/2017_icassp_reverberation.pdf http://danielpovey.com/files/2017_icassp_reverberation.pdf, or the ASPiRE model before) has a bigger impact on the practical usefulness of ASR than getting an X% relative improvement over the previous SOTA on the eval2000 set.