PetrisP's answer covered a lot of your question, but there are some additional considerations, even for English. Let's say you decide to grab all of those TV b

PetrisP's answer covered a lot of your question, but there are some additional considerations, even for English. Let's say you decide to grab all of those TV broadcasts and use them as training data (we'll ignore the fact that very little speech is as clean as the speech in TV broadcasts). Every mistake that occurs in the transcriptions you're using for training data represents the potential for mistraining your model when you perform a forced alignment [0]. Since most forced alignment tools perform a feature transformation step to increase how discriminative the phones are in feature space (ie how different they are from each other) and a step to reduce the number of dimensions, small errors in the transcriptions can result in huge differences in decoder behavior. Now factor in that your lexicon (the set of pronunciations you're using to map the orthographic transcriptions onto the possible phoneme strings that the speaker could have actually said) is almost certainly incomplete and very likely doesn't include nearly enough pronunciations for speakers with non-standard dialects or speakers with accents. Plus there are all sorts of typos in the original transcriptions; this often means that you need to run g2p to generate shitty pronunciations for words that actually don't exist, adding further errors into your model. Suddenly, you start thinking that all these decoder problems you're seeing are problems with the training data as it currently exists and the only solution is to do a phonetic transcription of the training data! You can still get that done, but since you've realized some of the blame lies with your lexicon you can't just dump the videos onto Mechanical Turk and have random people do them: you now need to get people that are familiar with phonetic transcription. And you decide that since even experts mess up a fair amount when transcribing speech, you're going to get 2 people to transcribe everything and throw away everything where they disagree. You aren't Google, so your training budget isn't that big. For each hour of speech you're spending 2 expert man hours and still throwing away 6 minutes and you need 10K+ hours of data at the bare minimum! And looking at the literature, it looks to you like there is a near linear correlation between who has the most data and who has the models that perform the best! So that's why people end up claiming that a fundamental problem with building out ASR is a lack of training data: because getting it and making it usable is very expensive, and because having lots of good training data let's you avoid addressing other difficult issues in ASR. [0] http://www1.icsi.berkeley.edu/Speech/faq/forcedalign.html http://www1.icsi.berkeley.edu/Speech/faq/forcedalign.html