3 ms·

Dunno why (probably dataset) but open source Speech Recognition models are performing very poorly on real world data compared to google speech to text or azure

Dunno why (probably dataset) but open source Speech Recognition models are performing very poorly on real world data compared to google speech to text or azure cognitive.

option 6y ago
The key is to fine-tune on your data. Take publicly available pretrained model, fine-tune on your data and you can often get results better than google’s service or azure cognitive on your use-case (google and azure asr are great general services, but they cannot do better than custom, say, health-center specific model)
dylanbfox 6y ago
One of the main factors for this is probably due to dataset size. Commercial STT models are trained on 10s of thousands of hours of data from real-world data. Even a decent model architecture is going to perform pretty well on that much data. Most open source models are trained on Libri, SWB, etc. which are not really big or diverse enough for real-world scenarios. But to max-out results the devil is in the details IMO (network architecture, optimizer, weight initialization, regularization, data augmentation, hyperparam tuning, etc) which requires a lot of experiments.
- bginsburg 6y ago
  There are new, very large public English speech datasets: Mozilla Common Voice, National Speech Corpus, which can be combined with LibriSpeech to train large models.
  - eindiran 6y ago
    If you combine them you get 5ish K hours of speech for English, which is still fairly small compared to what most big players have access to.
  - solidasparagus 6y ago
    Amazon worked with 7k hours of labeled data + 1 million hours of unlabeled data - https://arxiv.org/pdf/1904.01624.pdf https://arxiv.org/pdf/1904.01624.pdf
tasubotadas 6y ago
LibreSpeech is 1000h, WSJ is 73h. Google is training on datasets that are as big as 30kh and MS seems to work on a 10k h dataset. At the moment, I am working on a similar e2e system but 80h big dataset makes it a really challenging task to generalize well.

Dunno why (probably dataset) but open source Speech Recognition models are performing very poorly on real world data compared to google speech to text or azure cognitive.

option 6y ago

The key is to fine-tune on your data. Take publicly available pretrained model, fine-tune on your data and you can often get results better than google’s service or azure cognitive on your use-case (google and azure asr are great general services, but they cannot do better than custom, say, health-center specific model)

dylanbfox 6y ago

One of the main factors for this is probably due to dataset size. Commercial STT models are trained on 10s of thousands of hours of data from real-world data. Even a decent model architecture is going to perform pretty well on that much data. Most open source models are trained on Libri, SWB, etc. which are not really big or diverse enough for real-world scenarios. But to max-out results the devil is in the details IMO (network architecture, optimizer, weight initialization, regularization, data augmentation, hyperparam tuning, etc) which requires a lot of experiments.

bginsburg 6y ago
There are new, very large public English speech datasets: Mozilla Common Voice, National Speech Corpus, which can be combined with LibriSpeech to train large models.
- eindiran 6y ago
  If you combine them you get 5ish K hours of speech for English, which is still fairly small compared to what most big players have access to.
- solidasparagus 6y ago
  Amazon worked with 7k hours of labeled data + 1 million hours of unlabeled data - https://arxiv.org/pdf/1904.01624.pdf https://arxiv.org/pdf/1904.01624.pdf

tasubotadas 6y ago

LibreSpeech is 1000h, WSJ is 73h. Google is training on datasets that are as big as 30kh and MS seems to work on a 10k h dataset. At the moment, I am working on a similar e2e system but 80h big dataset makes it a really challenging task to generalize well.