3 ms·
There's lots of resources for English and much less for many other languages - this article is about the non-English case. Closed captioning and subtitles are
by PeterisP 7y ago
There's lots of resources for English and much less for many other languages - this article is about the non-English case.
Closed captioning and subtitles are often used, but they are 'dirty data' - they are usually not a one-to-one match, the differences mess up training. And licensing issues are a pain; "they aren't going to care about it" is not a solution and things like background music in movies (the rightholders very much do care about it, even if just to make a point) make it pretty much impossible for e.g. some university to legally distribute a dataset that includes movie audio tracks; so they can run some experiments on it themselves but as soon as you want any collaboration, that data is taboo.
Paying people to transcribe or read works, but it's not cheap. Reading 10000 hours takes at least 10000 hours of paid work and generally a bit more than that - mistakes matter, so you need review and correction. If you try transcribing things yourself, a tiny 100 hour dataset is going to take you at least something like 400 hours, which is months of work.
So that's the point - getting a usable dataset for some language costs hundreds of thousands of dollars if you're frugal, and millions if you want good results. And there are very many languages in the world.