3 ms·
Surely you can do much better than this using a speech recogniser?
by IshKebab 8y ago
Surely you can do much better than this using a speech recogniser?
- smacke 8y agoBy speech recognizer, are you referring to a method that would perform speech-to-text transcription? If so, you could probably do better when your video and subtitles are in the same language, but you would lose the ability to be language-agnostic (unless you manage to solve machine translation well enough :) Interestingly enough, I haven't actually found any cases where the synchronization doesn't work (assuming the only problem with the subtitles file is a time offset), so it actually looks like simple voice detection might be good enough for the target use-case of longer TV episodes and movies, although further evaluation is necessary.
- Axsuul 8y agoFrequently there will be subtitles that describe sounds and noises for the deaf (i.e. door closes). How is this handled?
- smacke 8y agoThe approach taken here is to not worry about those. :) These will introduce some noise, but the synchronization algorithm seems to be robust enough that noise doesn't matter (whether it's from the voice activity detector or from non speech-related subtitles).
- derefr 8y agoHow about if you just do speech-to-timed-phonemes (which doesn't require any kind of language model, just a model trained with people speaking in accents, to produce "canonical" phonemes) and text-to-phonemes (which just needs a dictionary with IPA metadata), and then attempt to synchronize your text phonemes to the audio's phonemes? It'd be the same as the current solution, but with higher dimensionality on the symbols.
- iforgotpassword 8y agoWhy? Detecting whether there is speech or not should be enough to align video and subs. It's not like the word order would be scrambled in the subtitle file and needs to be sorted according to the video. :)