3 ms·
What happens if the CC isn't entirely in sync with the video or audio?
by epmaybe 9y ago
What happens if the CC isn't entirely in sync with the video or audio?
- gwern 9y agoStrictly speaking, the text is never 'entirely in sync' because spoken words inherently blur together and are seamless; individual letters in the text do not start and end at precise intervals. This is one of the things that makes speech recognition so hard: letters, syllables, and words do not really exist as discrete things on the raw audio level. So this problem exists for any speech transcription dataset. To provide a loss function, then, you would use something like CTC: http://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=FD2D37798AA9ABA95F00F67FA604F1AD?doi=10.1.1.75.6306&rep=rep1&type=pdf http://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=FD2... Fortunately, NNs are good at handling noisy data, and in practice they work very well for speech recognition/transcription.
- eginhard 9y agoRecordings are force-aligned to the transcriptions anyway (using essentially a speech recognition system) to obtain phone-level alignments. You don't need explicit timing information beforehand.