25 ms·

Show HN: Automatically synchronize subtitles with video

583 points by smacke 8y ago

lainga 8y ago
Really neat! Because it doesn't rely on the video image data this would work for podcasts with subtitles (if they exist), right?
- smacke 8y ago
  Yes! it should work for this too, as long as ffmpeg recognizes the audio format :)
satori99 8y ago
This is cool. I am constantly impressed by ffmpeg and it capabilities. Last week I was able to make it act as a proxy for live streaming video that reduced the volume of ad-breaks by 50% automatically by listening for SCTE35 cue packets in the stream and adjusting a volume filter accordingly. My housemate has a reality TV addiction, and the ads were getting on my nerves. I just intended to see if it was possible, but I had a POC running in about 1/2 an hour.
- zeotroph 8y ago
  A neat subtitle feature I found out mpv has, but no other video player: Hide those parts of the subtitle which are mainly for those with impaired hearing like '[loud noise]' etc, but keep the rest: `mpv --sub-filter-sdh ...` That logic could probably be extracted into a separate subtitle filtering tool.
  - JorgeGT 8y ago
    If I remember correctly SubtitleEdit (https://github.com/SubtitleEdit/subtitleedit https://github.com/SubtitleEdit/subtitleedit) had such a tool for removing the hearing-impaired bits.
    - gsich 8y ago
      still has. I use it for every subtitle, gets rid of most common errors too.
  - aasasd 8y ago
    When going through Miami Vice I discovered that the SDH parts sometimes also act as notes for the culture-impaired. (Song titles and lyrics were a particularly nice touch, though not strictly SDH.) And for English-as-second-language folks, it's occasionally curious which expressions are used in the SDH descriptions.
- jakecopp 8y ago
  This sounds amazing! Did you use a USB TV tuner or an online source? Is there chance you could open source the code?
  - satori99 8y ago
    I used an online stream source; A HLS stream that one of my local TV stations broadcasts online. My method will only work with streams that have SCTE35 packets embedded. This is normally used for local ad insertion on cable networks, so it may not always be available on OTA streams. There is not much code because it is mostly ffmpeg that does the work. I used nodejs to manage an ffmpeg child process, read its output streams (one stream over a unix domain socket for the video, and the stdout stream for the SCTE35 packets) and pipe them to a HTTP client. My code parses the SCTE packets and then sends commands into the running ffmpeg child process to change the audio levels. I am happy to share it when I get home in a few hours.
    - deivid 8y ago
      please do
      - satori99 8y ago
        https://gist.github.com/satori99/2cb06938bfe8532ecc00e60906509052 https://gist.github.com/satori99/2cb06938bfe8532ecc00e609065...
    - jakecopp 8y ago
      Thanks so much!
  - satori99 8y ago
    https://gist.github.com/satori99/2cb06938bfe8532ecc00e60906509052 https://gist.github.com/satori99/2cb06938bfe8532ecc00e609065... Keep in mind it's PoC code, I stopped when it worked the first time :P
dearrifling 8y ago
Awesome! According to the description of the internals this only works with offsets and doesn't adjust the subtitle's playback speed. I think playback speed variations are not that broad, it's usually caused by playing a subtitle written for a given FPS played back on a different FPS video. It might worth it to try common FPS ratios at once and keep the best match.
- iforgotpassword 8y ago
  Yes that would be great, happens a lot when moving between pal and NTSC regions.
- aasasd 8y ago
  In the case when resulting subtitles turn out to be shorter than the input file, subtitle editing software like Aegisub can automatically trim ending times so they don't overlap with starting times. This could probably be similarly done in the script itself in the future. If the result gets longer and the times for each line are a bit too short―I guess you'll have to read faster, har har. --- Actually, iirc Aegisub can also do automatic stretching for different FPS, since that's a rather easy case. However, it's not enough when the subs are from a different media release or an edit―which made me long for exactly the kind of solution like in the post.
kozikow 8y ago
An even harder and cooler problem to solve would be finding extra scenes. Sometimes subtitle or movie have slightly different "cut" (e.g. directors cut) and there are some extra scenes added. A similar algorithmic approach could be used to solve this problem as well.
- smacke 8y ago
  Totally agreed. I'm not sure how to do the alignment scoring step in this case, perhaps some kind of DTW / FFT hybrid... definitely worth further consideration.
  - rcthompson 8y ago
    This sounds very similar to the problem of whole-genome alignment in bioinformatics. You have, say, the human and chimp genome sequences, and you want to align them to find out which portions of each genome correspond to which portions in the other, allowing for the possibility of insertions or deletions of DNA sequences in each one. Unfortunately, I'm not too familiar with the current state of the art for whole-genome alignment, so I don't know for sure which algorithms are considered, but MUMmer4[1] seems like a good place to start. These algorithms are designed to handle sequences up to billions of letters long (e.g. the human genome, which is about 3 billion letters). [1]: https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1005944 https://journals.plos.org/ploscompbiol/article?id=10.1371/jo...
lern_too_spel 8y ago
This problem is called forced alignment. YouTube's subtitler does it for you. If you don't want to upload your video and subtitles, here are some other tools that will work: https://github.com/pettarin/forced-alignment-tools/blob/master/README.md https://github.com/pettarin/forced-alignment-tools/blob/mast...
- smacke 8y ago
  Cool! It looks to me like most of those approaches are a lot more sophisticated and work even when the text doesn't have any time annotations. At a first glance, it looks like the major downside is that these approaches are geared toward particular target languages. Subsync's advantage is its simplicity: by limiting the acoustic analysis part to simple voice detection, it can perform alignment between subtitles and audio/video in different languages, which is great for when you want to watch a TV episode in another language, but your video includes the previous episode's recap and your subtitles do not (or vice-versa). I'll definitely be taking a deeper look at these to see if there's any ideas / functionality that can be further incorporated.
acqq 8y ago
"Although it can usually work if all you have is the video file, it will be faster (and potentially more accurate) if you have a correctly synchronized "reference" srt file, in which case you can do the following: subsync reference.srt -i unsynchronized.srt -o synchronized.srt" I really don't understand what the reference.srt is supposed to be here. Edit: If I have a "reference" which is synchronized why do I have to do anything at all? Why would I need to synchronize anything then? Edit2: If the French titles are a translation of the English ones, how wouldn't copying the timestamps from one to another be enough? Or do you want to say that the program can match the subtitles which aren't based on the English ones and have different number of sentences etc? But how can it then know which part of translation belongs to what? What's short in one language isn't in another etc.
- Outpox 8y ago
  I believe it's a typo and should read "Although it can usually work if all you have is the subtitles file [...]". This means that you would use a reference file that would allow you to synchronize your subtitles.
  - jmkb 8y ago
    I could imagine having a file with correctly-synced subs in the wrong language. Maybe that would be a good reference file. (I think you're correct subbing "subtitles file" for "video file".)
- smacke 8y ago
  Thanks for flagging -- this seems to be a common point of confusion; I'll work on updating the readme. Let's say reference.srt is a set of English subtitles, and you have an out-of-alignment French subtitles file unsynchronized.srt. You could use the video directly as a reference in order to synchronize the French subtitles, but it will take a bit longer since we need to extract audio and perform voice detection over the whole video file. In this case, it will be faster to use your already-synchronized English subtitles file reference.srt.
kojackst 8y ago
Sorry for the dumb question, but what exactly does this do? `subsync reference.srt -i unsynchronized.srt -o synchronized.srt` I mean, I already have a synchronized srt file, so what would I be syncing here?
- magissima 8y ago
  I'm guessing `synchronized.srt` is just the name of the output file.
  - smacke 8y ago
    I think they were referring to the fact that "reference.srt" would already be synchronized.
- smacke 8y ago
  Great question. The main use case I can think of is when reference.srt and unsynchronized.srt are in different languages, and you want to eventually merge them into a single dual-language subtitle file. EDIT: Oh, I should also mention that you don't need a reference.srt -- it can look at the video directly and use that as a reference.
  - zeotroph 8y ago
    Given it just processes audio, how long does processing one video file usually take?
    - smacke 8y ago
      Audio extraction is actually the most expensive part. It depends on the length of the video, but my experience is that it finishes in 20 to 30 seconds. It's possible one might be able to sample different parts of the video to bring the runtime down further; I plan to experiment with this when I get the chance.
  - kojackst 8y ago
    Ohh now I got you. I believe README should be improved in this part then. It reads: ==== Although it can usually work if all you have is the video file, it will be faster (and potentially more accurate) if you have a correctly synchronized "reference" srt file, in which case you can do the following: subsync reference.srt -i unsynchronized.srt -o synchronized.srt ==== I believe you should explain that if you have a reference file in another language which is correctly synchronized with that video, you can use that file instead of the video, as its timestamps will serve as references when synchronizing the target .srt file. Now this has raised a question, what if the reference file has a different block count? For example, in some languages (like Chinese or Japanese) we can say a lot with fewer characters than in English. So in Chinese a text will stay on the screen for a long time, whereas in English the corresponding text would be split into two or more blocks. Wouldn't that make synchronization less accurate? BTW that's a cool project. Thanks for sharing!
IshKebab 8y ago
Surely you can do much better than this using a speech recogniser?
- smacke 8y ago
  By speech recognizer, are you referring to a method that would perform speech-to-text transcription? If so, you could probably do better when your video and subtitles are in the same language, but you would lose the ability to be language-agnostic (unless you manage to solve machine translation well enough :) Interestingly enough, I haven't actually found any cases where the synchronization doesn't work (assuming the only problem with the subtitles file is a time offset), so it actually looks like simple voice detection might be good enough for the target use-case of longer TV episodes and movies, although further evaluation is necessary.
  - Axsuul 8y ago
    Frequently there will be subtitles that describe sounds and noises for the deaf (i.e. door closes). How is this handled?
    - smacke 8y ago
      The approach taken here is to not worry about those. :) These will introduce some noise, but the synchronization algorithm seems to be robust enough that noise doesn't matter (whether it's from the voice activity detector or from non speech-related subtitles).
  - derefr 8y ago
    How about if you just do speech-to-timed-phonemes (which doesn't require any kind of language model, just a model trained with people speaking in accents, to produce "canonical" phonemes) and text-to-phonemes (which just needs a dictionary with IPA metadata), and then attempt to synchronize your text phonemes to the audio's phonemes? It'd be the same as the current solution, but with higher dimensionality on the symbols.
- iforgotpassword 8y ago
anyfoo 8y ago
From the README: "[...] the naive O(n^2) strategy for scoring all alignments is unacceptable. Instead, we use the fact that "scoring all alignments" is a convolution operation and can be implemented with the Fast Fourier Transform (FFT), bringing the complexity down to O(n log n)." I absolutely love it when something that, at very first glance, has no business in being solved in the frequency domain, gets solved in the frequency domain.
- clairity 8y ago
  the fourier series was one of the things that blew my mind. any periodic function can be decomposed into a bunch of sine waves! even (the interesting portions of) many other functions can be approximated by a (potentially infinite) series of sine waves! it's simply madness i tell you.
  - scarmig 8y ago
    Interesting bit of history, Kolmogorov became famous when he published his first scientific paper (at 19!) on the construction of a function whose Fourier series diverges (almost) everywhere.
- hosay123 8y ago
  I thought I understood FFT, but totally not getting the relationship to this problem. Someone ELI5 please? :)
  - Iburinoc 8y ago
    The problem being solved essentially is: you have two binary strings, and you want to offset one of them so that they match up the best. For each offset, you're taking a dot product of one sequence with the offset version of the other. This is the same as computing the convolution of the two sequences together (https://en.wikipedia.org/wiki/Convolution https://en.wikipedia.org/wiki/Convolution). Computing this naively would be O(n^2) (doing linear work for each possible offset). One property of the Fourier transform is that convolution in the time domain corresponds to element-wise multiplication in the frequency domain (https://en.wikipedia.org/wiki/Convolution_theorem https://en.wikipedia.org/wiki/Convolution_theorem), so you can compute the convolution efficiently by taking the FFT of both series, doing element-wise multiplication, and then taking the inverse FFT of the result.
symstym 8y ago
Nice! Have you seen this similar project? If not sure if there are any ideas to be borrowed, but it sounds like it achieved excellent accuracy: https://github.com/AlbertoSabater/subtitle-synchronization https://github.com/AlbertoSabater/subtitle-synchronization
- smacke 8y ago
  Interesting! I was not aware of that project before. It looks like it could be worthwhile to incorporate their neural net-based VAD, if it's not too slow. At a first glance, it looks like the main difference is the postprocessing step -- they use some heuristics to avoid trying all possible alignments, while subsync uses FFT as its secret sauce to get away with trying out all of them. :) I should include in the readme that for >1 hour movies, it usually finishes the synchronization in ~20 seconds, which compares favorably to the project linked in your comment (13 minutes unoptimized, 2 minutes optimized).
loeg 8y ago
> I have yet to find a case where the automatic synchronization has been off by more than ~1 second That's actually quite bad synchronization error for A-V; maybe it's less noticeable for subtitles. I wonder if there is a good way to reduce that error down to milliseconds.
- smacke 8y ago
  It's actually fairly annoying for subtitles, but it's useful for getting videos in unfamiliar languages to the right ballpark. Afterwards, my go-to so far has been to just perform a manual one-off adjustment in VLC after the video starts (you can offset subtitles in increments / decrements of 50ms with hotkeys h and j).
- gsich 8y ago
  100 ms A/V is already quite noticeable, with some training one might spot 50 ms too.
  - loeg 8y ago
    I agree 100ms is bad :-). I think even 10s of ms sync error is readily jarring for most people, at least for A-V. Subtitles may be more forgiving.
Theodores 8y ago
Subtitles are in the realm of not many people doing interesting things with them, which is a pity as they are such a creative medium. Recently I rolled my own code to play WebVTT to an audio (think video without the pictures) track. I had assumed there would be off the shelf libraries that would do this for me. Oh how wrong I was! I took it a bit further than white text in a black box at the foot of the screen. Not having pictures kind of made it that way. So I decided I needed cartoon grade speech bubbles, with the speech bubble coming from the left for one voice and the other side of the screen for the other voice, again finding myself in the realms of no ready made examples to do this. The speech bubbles had to scale to fit the content with them being suitably rounded, a la cartoon style. I found an SVG solution to my problem. The WebVTT format and variants have all kinds of goodies in them to position speech and do things in time, as per karaoke. I think one reason I found myself in the world of rare code was that we assume anything to do with accessibility is for disabled people and therefore has to be boring and done with no creativity whatsoever.
robert-boehnke 8y ago
> Instead, we use the fact that "scoring all alignments" is a convolution operation and can be implemented with the Fast Fourier Transform (FFT), bringing the complexity down to O(n log n). Not sure I understand this right - is this basically treating both binary strings as square waves, converting them to the frequency domain and determining the offset as a pitch shift between the two spectrograms?
- smacke 8y ago
  Precisely. If I could more easily get at the raw classifier output of webrtcvad, it should be possible to be even smarter (we could have square waves with any amplitude between -1 and +1, not just either -1 and +1, which should take into account the classifier uncertainty). EDIT: err, I'm actually not sure about the pitch shift part, that's a bit of vocabulary I'm not familiar with. If you've seen the fast polynomial multiplication algo from CLRS, it's basically that. E.g. if we have strings 1101 and 0101, we can find the best alignment by looking at the exponent of the largest coefficient after multiplying polynomial(1101)*polynomial(reverse(0101)) where polynomial(1101) = x^3 + x^2 - x + 1 and polynomial(reverse(0101)) = polynomial(1010) = x^3 - x^2 + x - 1
- rocqua 8y ago
  That is a very 'signals' based interpretation of the convolution theorem [1] applied to binary functions. In essence, the fourier-transform is based on convolutions. F(eta) is essentially the convolution of f(x) with sin(eta x).* This is very loosely why the convolution theorem works. [1] https://en.wikipedia.org/wiki/Convolution_theorem https://en.wikipedia.org/wiki/Convolution_theorem * This excludes all cosine parts of the transform. Its neater to work in the complex domain and state that: F(eta) = f(x) convolved with e^(eta i x)
ummonk 8y ago
I'm still waiting for the day there is an automatic synchronizer for the video and audio tracks in VLC. It's so frustrating trying to find the right offset when they get out of sync.
draz 8y ago
This is an area I am actively looking into at the moment for my employer. It's an interesting project, and I am excited to see work in this domain. Note that this will not pass QC at any broadcaster/OTT. Our tolerance is typically within 1-12 frames (and a bunch of other requirements around shot changes, etc.)
- barrystaes 8y ago
  > Our tolerance is typically within 1-12 frames (and a bunch of other requirements around shot changes, etc.) Tell me more!
  - draz 8y ago
    Netflix did a good job outlining their requirements (which are mirroring industry requirements): https://partnerhelp.netflixstudios.com/hc/en-us/articles/215758617-Timed-Text-Style-Guide-General-Requirements https://partnerhelp.netflixstudios.com/hc/en-us/articles/215... Note that I do not work for Netflix rather another multinational mass media conglomerate.
chanux 8y ago
I remember the annoyance. I created a script to eye ball, guess and fix back then. It was good enough. https://gist.github.com/chanux/2042676 https://gist.github.com/chanux/2042676
psychometry 8y ago
You should look into DNA and RNA sequence alignment algos for possible improvements on your FFT. It's a huge and very well-explored space.
- zimpenfish 8y ago
  Since I already had it open (for unrelated reasons), this might be relevant. https://en.wikipedia.org/wiki/Smith%E2%80%93Waterman_algorithm https://en.wikipedia.org/wiki/Smith%E2%80%93Waterman_algorit... > The Smith–Waterman algorithm performs local sequence alignment; that is, for determining similar regions between two strings of nucleic acid sequences or protein sequences.
  - amelius 8y ago
    But also: > The Smith–Waterman algorithm is fairly demanding of time: To align two sequences of lengths m and n, O(mn) time is required.
    - zimpenfish 8y ago
      Thankfully I'm only working with short sequences but yeah, I guess that might be a problem.
- smacke 8y ago
  Agreed -- definitely worth thoroughly investigating this space to see whether there are any ideas that can be borrowed.
kumarharsh 8y ago
This blows my mind (like everyone else's here)! While playing digital copies of somewhat older movies (before BluRay rips came into vogue), a problem which surfaces frequently is that the frame rates of the video and the subtitle track are slightly mismatched, say video at 24fps, subtitle at 25.6fps(?). This makes the subtitles drift away from the video and require a manual intervention every few minutes. If I can't hear the audio properly for some reason, then I just become fed up and don't watch the movie at all. Add to that the existence of different 'cuts' for films, which add another dimension to the subtitle problem. How would one even go about solving this problem?
- xfer 8y ago
  If you use vlc there is a subtitle fps that you can modify.
- rocqua 8y ago
  As others have said, matching cuts is very similar to genome matching. In genome matching, often genes are slightly offset, so there is need to correct for that. Matching speed might be possible based on first matching runs of one minute, and then trying to scale with that minute as a fixed point to see which scale works out. Perhaps some heuristics to take common frame rate-ratios into account might help. An even crazier idea: Use the fact that scaling becomes shifting after a logarithm, and try to match after setting g(x) = f(log(x)).
kumarharsh 8y ago
Completely tangential question to this awesome discussion: who even writes subtitles? It feels like a thankless job to write subtitles for rips of movies and TV shows old and new. I get that maybe they source original from, say, Netflix, but .set files existed long before Netflix, and for lots of movies not on Netflix. Writing all those tags for hearing impaired seems like a lot of work, which anyone other than the first-person movie production team would be loathe to indulge in. Yet I see a lot of subtitles which don't seem 'official'. Then there are subtitles which seem like they were written as a loose translation of the audio. Both are in English, so I don't see why they can't just transcribe what's being said. Instead, for dialogues such as "The greatest trick the Devil ever pulled was convincing the world he didn't exist.", the subtitle is written as "The devil tricked the world into thinking he didn't exist."
- ubercow13 8y ago
  People actually _pay_ to watch TV shows, so imagine being paid to do it!
- roddds 8y ago
  I'm not super familiar with the specifics and I'm sure someone who's worked on this particular probably will be able to clarify, but the way I understand it there is a limit on how many characters can be displayed on the screen at any given time, as well as a minimum amount of time a subtitle should be visible on the screen. Because of this, transcribers might be forced to write down a shorter version of the line in order to make it fit.
- kakarot 8y ago
  Thankfully are jobs will be getting easier as speech recognition continues to improve. Some people start from scratch, some people find existing hard subs and translate them to soft subs, some people just translate subs, etc... some people are prolific but the reason it seems most media has a sub is that there are millions of interested and capable people out there, so if a movie is only meaningful enough to only one person out of say 40 million, that's still one person who will make subs for that movie. We have the countless masses to thank, even when sometimes the quality isn't perfect.
- 8y ago
sidcool 8y ago
This is quite amazing. The code quality overall looks pretty good too. Pleasantly surprised to see it in Python. Was dreading complex C++ code. Thanks for sharing. The code will aid my learning.
joshstrange 8y ago
This looks pretty cool, right now I drop down to the command line to shift subs either direction but if they are off by too much or commercials/breaks don't line up then it's game over. I'll have to drop this out on my server, I'm tempted to run it across everything but I'll probably settle for running it manually when something doesn't line up in plex. Edit: Hmm, my first test moved the subtitles but they ended up 3-5 seconds after the audio instead of the original 2-5 before... I'll have to keep testing when I come across bad subs.
- smacke 8y ago
  My guess is that your subtitles don't have a "fixed drift". Unfortunately, this project only works for constant drift at the moment -- dynamic subtitle drift is a harder problem but absolutely worth thinking more about for future work.
hendry 8y ago
What about syncing up audio? Yet to find a CLI solution for syncing up & replacing external audio on my camera's MP4. Current solution is boot up FCPX and do a sync there which rips away minutes of my life.
- gsich 8y ago
  I use Audacity for this. I load both files, look for a peak or end of silence in both. The difference I remux with Mkvtoolnix. I do this with 2-3 spots. Loading the file is usually the longest waiting period.
bbx 8y ago
Thanks for this! I just happen to need it. I have the first season of my favourite US series, but dubbed in Spanish, because I want to train my ear. Unfortunately the subtitle files slowly drift apart from the audio by 10 seconds during each episode. I used to have a script to “stretch” subtitles manually, but it’s quite fiddly: https://gist.github.com/jgthms/7dfc20db3478a938069a0191c4e30843 https://gist.github.com/jgthms/7dfc20db3478a938069a0191c4e30...
- smacke 8y ago
  You are most welcome, although unfortunately it looks like this project doesn't support your use case yet. Right now it can only correct "constant drift", although I'll definitely be thinking more about other scenarios based on the reception. That said, it sounds like for your particular case, if you use VLC, you might be able to adjust the subtitle FPS so that you don't have to continuously manually resync. (I haven't run into this so just going off what others have mentioned.)
hantusk 8y ago
I've had great results with the Descript app which can both transcribe and can export a synchronized subtitle file. https://www.descript.com/ https://www.descript.com/ Perhaps you can import an srt file, and have it sync'ed? or perhaps you can export the srt text and let Descript sync it and re-create an .srt file.
aorth 8y ago
Super cool. This and the Python-based subtitle downloader subliminal make working with subtitles really easy. I had previously relied on incremental fudging of times using trial and error with www.sync-subtitles.com.
kowdermeister 8y ago
> I have yet to find a case where the automatic synchronization has been off by more than ~1 second. 1 second is in the 0 - 1000ms which is still a lot of room for manual tweaking in VLC. I can notice 50ms differences in sync.
EvagohP5 8y ago
I will love you for this. really. I can't tell how much time I have lost searching for the correct subtitles, in various languages
apexalpha 8y ago
Please tell me someone that works at Plex is reading this thread. I know it might seem a bit distant for Americans but this HUGE!
- Guillaume86 8y ago
  A Kodi plugin might be a more realistic target, subtitles handling in Kodi is pretty great in my experience, especially compared to Plex (download while playing, sources addons, offset adjustement).
boolemancer 8y ago
I recently ran across a similar tool for aligning subtitles based on a reference subtitle file called aligner: https://github.com/kaegi/aligner https://github.com/kaegi/aligner It seemed to work pretty well in my testing. Maybe it could be useful to look at?
- smacke 8y ago
  This project came up when I was researching this idea. It looks especially useful because the alignment algorithm is less restrictive (can introduce splits/breaks), so I plan on taking a closer look for sure. Given that it does not do any speech detection and requires a reference subtitles file, I'm not sure how well it would work for aligning the encoding for the speech-induced part, which will have a lot more noise. Particularly, the linked project works by aligning time intervals [a,b], while subsync is "dumber" in that it discretizes things into binary strings, and an interval constructed by merging consecutive 1's and 0's in speech detection-induced binary strings will probably be a lot harder to match to the subtitle intervals. Either way, I also thought this was a cool project and will be thinking more about whether any ideas can be borrowed.
s0adex 8y ago
It seems people are going back to P2P for movies. I'd forgotten the ordeals of unsynchronized srt files. Thanks to the Netflix era.
- Krasnol 8y ago
  Probably because not everything you want to see is on Netflix anymore and it's getting worse.
hobbes78 8y ago
I see this will only shift the timings some offset. But many times it isn't enough e.g. if the video source is NTSC and the subtitle were meant for PAL. In this scenario I use Subtitle Workshop, which allows me to specify two (or more) pairs of times, and all subtitles are synchronized accordingly. Optimally the two pairs of times would be near the beginning and the end, but to prevent big plot spoilers, I usually use the beginning and 2/3 of the length.

583 points by smacke 8y ago

Really neat! Because it doesn't rely on the video image data this would work for podcasts with subtitles (if they exist), right?

smacke 8y ago
Yes! it should work for this too, as long as ffmpeg recognizes the audio format :)

satori99 8y ago

This is cool. I am constantly impressed by ffmpeg and it capabilities. Last week I was able to make it act as a proxy for live streaming video that reduced the volume of ad-breaks by 50% automatically by listening for SCTE35 cue packets in the stream and adjusting a volume filter accordingly. My housemate has a reality TV addiction, and the ads were getting on my nerves. I just intended to see if it was possible, but I had a POC running in about 1/2 an hour.

zeotroph 8y ago
A neat subtitle feature I found out mpv has, but no other video player: Hide those parts of the subtitle which are mainly for those with impaired hearing like '[loud noise]' etc, but keep the rest: `mpv --sub-filter-sdh ...` That logic could probably be extracted into a separate subtitle filtering tool.
- JorgeGT 8y ago
  If I remember correctly SubtitleEdit (https://github.com/SubtitleEdit/subtitleedit https://github.com/SubtitleEdit/subtitleedit) had such a tool for removing the hearing-impaired bits.
  - gsich 8y ago
    still has. I use it for every subtitle, gets rid of most common errors too.
- aasasd 8y ago
  When going through Miami Vice I discovered that the SDH parts sometimes also act as notes for the culture-impaired. (Song titles and lyrics were a particularly nice touch, though not strictly SDH.) And for English-as-second-language folks, it's occasionally curious which expressions are used in the SDH descriptions.
jakecopp 8y ago
This sounds amazing! Did you use a USB TV tuner or an online source? Is there chance you could open source the code?
- satori99 8y ago
  I used an online stream source; A HLS stream that one of my local TV stations broadcasts online. My method will only work with streams that have SCTE35 packets embedded. This is normally used for local ad insertion on cable networks, so it may not always be available on OTA streams. There is not much code because it is mostly ffmpeg that does the work. I used nodejs to manage an ffmpeg child process, read its output streams (one stream over a unix domain socket for the video, and the stdout stream for the SCTE35 packets) and pipe them to a HTTP client. My code parses the SCTE packets and then sends commands into the running ffmpeg child process to change the audio levels. I am happy to share it when I get home in a few hours.
  - deivid 8y ago
    please do
    - satori99 8y ago
      https://gist.github.com/satori99/2cb06938bfe8532ecc00e60906509052 https://gist.github.com/satori99/2cb06938bfe8532ecc00e609065...
  - jakecopp 8y ago
    Thanks so much!
- satori99 8y ago
  https://gist.github.com/satori99/2cb06938bfe8532ecc00e60906509052 https://gist.github.com/satori99/2cb06938bfe8532ecc00e609065... Keep in mind it's PoC code, I stopped when it worked the first time :P

dearrifling 8y ago

Awesome! According to the description of the internals this only works with offsets and doesn't adjust the subtitle's playback speed. I think playback speed variations are not that broad, it's usually caused by playing a subtitle written for a given FPS played back on a different FPS video. It might worth it to try common FPS ratios at once and keep the best match.

iforgotpassword 8y ago
Yes that would be great, happens a lot when moving between pal and NTSC regions.
aasasd 8y ago
In the case when resulting subtitles turn out to be shorter than the input file, subtitle editing software like Aegisub can automatically trim ending times so they don't overlap with starting times. This could probably be similarly done in the script itself in the future. If the result gets longer and the times for each line are a bit too short―I guess you'll have to read faster, har har. --- Actually, iirc Aegisub can also do automatic stretching for different FPS, since that's a rather easy case. However, it's not enough when the subs are from a different media release or an edit―which made me long for exactly the kind of solution like in the post.

kozikow 8y ago

An even harder and cooler problem to solve would be finding extra scenes. Sometimes subtitle or movie have slightly different "cut" (e.g. directors cut) and there are some extra scenes added. A similar algorithmic approach could be used to solve this problem as well.

smacke 8y ago
Totally agreed. I'm not sure how to do the alignment scoring step in this case, perhaps some kind of DTW / FFT hybrid... definitely worth further consideration.
- rcthompson 8y ago
  This sounds very similar to the problem of whole-genome alignment in bioinformatics. You have, say, the human and chimp genome sequences, and you want to align them to find out which portions of each genome correspond to which portions in the other, allowing for the possibility of insertions or deletions of DNA sequences in each one. Unfortunately, I'm not too familiar with the current state of the art for whole-genome alignment, so I don't know for sure which algorithms are considered, but MUMmer4[1] seems like a good place to start. These algorithms are designed to handle sequences up to billions of letters long (e.g. the human genome, which is about 3 billion letters). [1]: https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1005944 https://journals.plos.org/ploscompbiol/article?id=10.1371/jo...

lern_too_spel 8y ago

This problem is called forced alignment. YouTube's subtitler does it for you. If you don't want to upload your video and subtitles, here are some other tools that will work: https://github.com/pettarin/forced-alignment-tools/blob/master/README.md https://github.com/pettarin/forced-alignment-tools/blob/mast...

smacke 8y ago
Cool! It looks to me like most of those approaches are a lot more sophisticated and work even when the text doesn't have any time annotations. At a first glance, it looks like the major downside is that these approaches are geared toward particular target languages. Subsync's advantage is its simplicity: by limiting the acoustic analysis part to simple voice detection, it can perform alignment between subtitles and audio/video in different languages, which is great for when you want to watch a TV episode in another language, but your video includes the previous episode's recap and your subtitles do not (or vice-versa). I'll definitely be taking a deeper look at these to see if there's any ideas / functionality that can be further incorporated.

acqq 8y ago

"Although it can usually work if all you have is the video file, it will be faster (and potentially more accurate) if you have a correctly synchronized "reference" srt file, in which case you can do the following: subsync reference.srt -i unsynchronized.srt -o synchronized.srt" I really don't understand what the reference.srt is supposed to be here. Edit: If I have a "reference" which is synchronized why do I have to do anything at all? Why would I need to synchronize anything then? Edit2: If the French titles are a translation of the English ones, how wouldn't copying the timestamps from one to another be enough? Or do you want to say that the program can match the subtitles which aren't based on the English ones and have different number of sentences etc? But how can it then know which part of translation belongs to what? What's short in one language isn't in another etc.

Outpox 8y ago
I believe it's a typo and should read "Although it can usually work if all you have is the subtitles file [...]". This means that you would use a reference file that would allow you to synchronize your subtitles.
- jmkb 8y ago
  I could imagine having a file with correctly-synced subs in the wrong language. Maybe that would be a good reference file. (I think you're correct subbing "subtitles file" for "video file".)
smacke 8y ago
Thanks for flagging -- this seems to be a common point of confusion; I'll work on updating the readme. Let's say reference.srt is a set of English subtitles, and you have an out-of-alignment French subtitles file unsynchronized.srt. You could use the video directly as a reference in order to synchronize the French subtitles, but it will take a bit longer since we need to extract audio and perform voice detection over the whole video file. In this case, it will be faster to use your already-synchronized English subtitles file reference.srt.

kojackst 8y ago

Sorry for the dumb question, but what exactly does this do? `subsync reference.srt -i unsynchronized.srt -o synchronized.srt` I mean, I already have a synchronized srt file, so what would I be syncing here?

magissima 8y ago
I'm guessing `synchronized.srt` is just the name of the output file.
- smacke 8y ago
  I think they were referring to the fact that "reference.srt" would already be synchronized.
smacke 8y ago
Great question. The main use case I can think of is when reference.srt and unsynchronized.srt are in different languages, and you want to eventually merge them into a single dual-language subtitle file. EDIT: Oh, I should also mention that you don't need a reference.srt -- it can look at the video directly and use that as a reference.
- zeotroph 8y ago
  Given it just processes audio, how long does processing one video file usually take?
  - smacke 8y ago
    Audio extraction is actually the most expensive part. It depends on the length of the video, but my experience is that it finishes in 20 to 30 seconds. It's possible one might be able to sample different parts of the video to bring the runtime down further; I plan to experiment with this when I get the chance.
- kojackst 8y ago
  Ohh now I got you. I believe README should be improved in this part then. It reads: ==== Although it can usually work if all you have is the video file, it will be faster (and potentially more accurate) if you have a correctly synchronized "reference" srt file, in which case you can do the following: subsync reference.srt -i unsynchronized.srt -o synchronized.srt ==== I believe you should explain that if you have a reference file in another language which is correctly synchronized with that video, you can use that file instead of the video, as its timestamps will serve as references when synchronizing the target .srt file. Now this has raised a question, what if the reference file has a different block count? For example, in some languages (like Chinese or Japanese) we can say a lot with fewer characters than in English. So in Chinese a text will stay on the screen for a long time, whereas in English the corresponding text would be split into two or more blocks. Wouldn't that make synchronization less accurate? BTW that's a cool project. Thanks for sharing!

IshKebab 8y ago

Surely you can do much better than this using a speech recogniser?

smacke 8y ago
By speech recognizer, are you referring to a method that would perform speech-to-text transcription? If so, you could probably do better when your video and subtitles are in the same language, but you would lose the ability to be language-agnostic (unless you manage to solve machine translation well enough :) Interestingly enough, I haven't actually found any cases where the synchronization doesn't work (assuming the only problem with the subtitles file is a time offset), so it actually looks like simple voice detection might be good enough for the target use-case of longer TV episodes and movies, although further evaluation is necessary.
- Axsuul 8y ago
  Frequently there will be subtitles that describe sounds and noises for the deaf (i.e. door closes). How is this handled?
  - smacke 8y ago
    The approach taken here is to not worry about those. :) These will introduce some noise, but the synchronization algorithm seems to be robust enough that noise doesn't matter (whether it's from the voice activity detector or from non speech-related subtitles).
- derefr 8y ago
  How about if you just do speech-to-timed-phonemes (which doesn't require any kind of language model, just a model trained with people speaking in accents, to produce "canonical" phonemes) and text-to-phonemes (which just needs a dictionary with IPA metadata), and then attempt to synchronize your text phonemes to the audio's phonemes? It'd be the same as the current solution, but with higher dimensionality on the symbols.
iforgotpassword 8y ago

anyfoo 8y ago

From the README: "[...] the naive O(n^2) strategy for scoring all alignments is unacceptable. Instead, we use the fact that "scoring all alignments" is a convolution operation and can be implemented with the Fast Fourier Transform (FFT), bringing the complexity down to O(n log n)." I absolutely love it when something that, at very first glance, has no business in being solved in the frequency domain, gets solved in the frequency domain.

clairity 8y ago
the fourier series was one of the things that blew my mind. any periodic function can be decomposed into a bunch of sine waves! even (the interesting portions of) many other functions can be approximated by a (potentially infinite) series of sine waves! it's simply madness i tell you.
- scarmig 8y ago
  Interesting bit of history, Kolmogorov became famous when he published his first scientific paper (at 19!) on the construction of a function whose Fourier series diverges (almost) everywhere.
hosay123 8y ago
I thought I understood FFT, but totally not getting the relationship to this problem. Someone ELI5 please? :)
- Iburinoc 8y ago
  The problem being solved essentially is: you have two binary strings, and you want to offset one of them so that they match up the best. For each offset, you're taking a dot product of one sequence with the offset version of the other. This is the same as computing the convolution of the two sequences together (https://en.wikipedia.org/wiki/Convolution https://en.wikipedia.org/wiki/Convolution). Computing this naively would be O(n^2) (doing linear work for each possible offset). One property of the Fourier transform is that convolution in the time domain corresponds to element-wise multiplication in the frequency domain (https://en.wikipedia.org/wiki/Convolution_theorem https://en.wikipedia.org/wiki/Convolution_theorem), so you can compute the convolution efficiently by taking the FFT of both series, doing element-wise multiplication, and then taking the inverse FFT of the result.

symstym 8y ago

Nice! Have you seen this similar project? If not sure if there are any ideas to be borrowed, but it sounds like it achieved excellent accuracy: https://github.com/AlbertoSabater/subtitle-synchronization https://github.com/AlbertoSabater/subtitle-synchronization

smacke 8y ago
Interesting! I was not aware of that project before. It looks like it could be worthwhile to incorporate their neural net-based VAD, if it's not too slow. At a first glance, it looks like the main difference is the postprocessing step -- they use some heuristics to avoid trying all possible alignments, while subsync uses FFT as its secret sauce to get away with trying out all of them. :) I should include in the readme that for >1 hour movies, it usually finishes the synchronization in ~20 seconds, which compares favorably to the project linked in your comment (13 minutes unoptimized, 2 minutes optimized).

loeg 8y ago

> I have yet to find a case where the automatic synchronization has been off by more than ~1 second That's actually quite bad synchronization error for A-V; maybe it's less noticeable for subtitles. I wonder if there is a good way to reduce that error down to milliseconds.

smacke 8y ago
It's actually fairly annoying for subtitles, but it's useful for getting videos in unfamiliar languages to the right ballpark. Afterwards, my go-to so far has been to just perform a manual one-off adjustment in VLC after the video starts (you can offset subtitles in increments / decrements of 50ms with hotkeys h and j).
gsich 8y ago
100 ms A/V is already quite noticeable, with some training one might spot 50 ms too.
- loeg 8y ago
  I agree 100ms is bad :-). I think even 10s of ms sync error is readily jarring for most people, at least for A-V. Subtitles may be more forgiving.

Theodores 8y ago

Subtitles are in the realm of not many people doing interesting things with them, which is a pity as they are such a creative medium. Recently I rolled my own code to play WebVTT to an audio (think video without the pictures) track. I had assumed there would be off the shelf libraries that would do this for me. Oh how wrong I was! I took it a bit further than white text in a black box at the foot of the screen. Not having pictures kind of made it that way. So I decided I needed cartoon grade speech bubbles, with the speech bubble coming from the left for one voice and the other side of the screen for the other voice, again finding myself in the realms of no ready made examples to do this. The speech bubbles had to scale to fit the content with them being suitably rounded, a la cartoon style. I found an SVG solution to my problem. The WebVTT format and variants have all kinds of goodies in them to position speech and do things in time, as per karaoke. I think one reason I found myself in the world of rare code was that we assume anything to do with accessibility is for disabled people and therefore has to be boring and done with no creativity whatsoever.

robert-boehnke 8y ago

> Instead, we use the fact that "scoring all alignments" is a convolution operation and can be implemented with the Fast Fourier Transform (FFT), bringing the complexity down to O(n log n). Not sure I understand this right - is this basically treating both binary strings as square waves, converting them to the frequency domain and determining the offset as a pitch shift between the two spectrograms?

smacke 8y ago
Precisely. If I could more easily get at the raw classifier output of webrtcvad, it should be possible to be even smarter (we could have square waves with any amplitude between -1 and +1, not just either -1 and +1, which should take into account the classifier uncertainty). EDIT: err, I'm actually not sure about the pitch shift part, that's a bit of vocabulary I'm not familiar with. If you've seen the fast polynomial multiplication algo from CLRS, it's basically that. E.g. if we have strings 1101 and 0101, we can find the best alignment by looking at the exponent of the largest coefficient after multiplying polynomial(1101)*polynomial(reverse(0101)) where polynomial(1101) = x^3 + x^2 - x + 1 and polynomial(reverse(0101)) = polynomial(1010) = x^3 - x^2 + x - 1
rocqua 8y ago
That is a very 'signals' based interpretation of the convolution theorem [1] applied to binary functions. In essence, the fourier-transform is based on convolutions. F(eta) is essentially the convolution of f(x) with sin(eta x).* This is very loosely why the convolution theorem works. [1] https://en.wikipedia.org/wiki/Convolution_theorem https://en.wikipedia.org/wiki/Convolution_theorem * This excludes all cosine parts of the transform. Its neater to work in the complex domain and state that: F(eta) = f(x) convolved with e^(eta i x)

ummonk 8y ago

I'm still waiting for the day there is an automatic synchronizer for the video and audio tracks in VLC. It's so frustrating trying to find the right offset when they get out of sync.

draz 8y ago

This is an area I am actively looking into at the moment for my employer. It's an interesting project, and I am excited to see work in this domain. Note that this will not pass QC at any broadcaster/OTT. Our tolerance is typically within 1-12 frames (and a bunch of other requirements around shot changes, etc.)

barrystaes 8y ago
> Our tolerance is typically within 1-12 frames (and a bunch of other requirements around shot changes, etc.) Tell me more!
- draz 8y ago
  Netflix did a good job outlining their requirements (which are mirroring industry requirements): https://partnerhelp.netflixstudios.com/hc/en-us/articles/215758617-Timed-Text-Style-Guide-General-Requirements https://partnerhelp.netflixstudios.com/hc/en-us/articles/215... Note that I do not work for Netflix rather another multinational mass media conglomerate.

chanux 8y ago

I remember the annoyance. I created a script to eye ball, guess and fix back then. It was good enough. https://gist.github.com/chanux/2042676 https://gist.github.com/chanux/2042676

psychometry 8y ago

You should look into DNA and RNA sequence alignment algos for possible improvements on your FFT. It's a huge and very well-explored space.

zimpenfish 8y ago
Since I already had it open (for unrelated reasons), this might be relevant. https://en.wikipedia.org/wiki/Smith%E2%80%93Waterman_algorithm https://en.wikipedia.org/wiki/Smith%E2%80%93Waterman_algorit... > The Smith–Waterman algorithm performs local sequence alignment; that is, for determining similar regions between two strings of nucleic acid sequences or protein sequences.
- amelius 8y ago
  But also: > The Smith–Waterman algorithm is fairly demanding of time: To align two sequences of lengths m and n, O(mn) time is required.
  - zimpenfish 8y ago
    Thankfully I'm only working with short sequences but yeah, I guess that might be a problem.
smacke 8y ago
Agreed -- definitely worth thoroughly investigating this space to see whether there are any ideas that can be borrowed.

kumarharsh 8y ago

This blows my mind (like everyone else's here)! While playing digital copies of somewhat older movies (before BluRay rips came into vogue), a problem which surfaces frequently is that the frame rates of the video and the subtitle track are slightly mismatched, say video at 24fps, subtitle at 25.6fps(?). This makes the subtitles drift away from the video and require a manual intervention every few minutes. If I can't hear the audio properly for some reason, then I just become fed up and don't watch the movie at all. Add to that the existence of different 'cuts' for films, which add another dimension to the subtitle problem. How would one even go about solving this problem?

xfer 8y ago
If you use vlc there is a subtitle fps that you can modify.
rocqua 8y ago
As others have said, matching cuts is very similar to genome matching. In genome matching, often genes are slightly offset, so there is need to correct for that. Matching speed might be possible based on first matching runs of one minute, and then trying to scale with that minute as a fixed point to see which scale works out. Perhaps some heuristics to take common frame rate-ratios into account might help. An even crazier idea: Use the fact that scaling becomes shifting after a logarithm, and try to match after setting g(x) = f(log(x)).

kumarharsh 8y ago

Completely tangential question to this awesome discussion: who even writes subtitles? It feels like a thankless job to write subtitles for rips of movies and TV shows old and new. I get that maybe they source original from, say, Netflix, but .set files existed long before Netflix, and for lots of movies not on Netflix. Writing all those tags for hearing impaired seems like a lot of work, which anyone other than the first-person movie production team would be loathe to indulge in. Yet I see a lot of subtitles which don't seem 'official'. Then there are subtitles which seem like they were written as a loose translation of the audio. Both are in English, so I don't see why they can't just transcribe what's being said. Instead, for dialogues such as "The greatest trick the Devil ever pulled was convincing the world he didn't exist.", the subtitle is written as "The devil tricked the world into thinking he didn't exist."

ubercow13 8y ago
People actually _pay_ to watch TV shows, so imagine being paid to do it!
roddds 8y ago
I'm not super familiar with the specifics and I'm sure someone who's worked on this particular probably will be able to clarify, but the way I understand it there is a limit on how many characters can be displayed on the screen at any given time, as well as a minimum amount of time a subtitle should be visible on the screen. Because of this, transcribers might be forced to write down a shorter version of the line in order to make it fit.
kakarot 8y ago
Thankfully are jobs will be getting easier as speech recognition continues to improve. Some people start from scratch, some people find existing hard subs and translate them to soft subs, some people just translate subs, etc... some people are prolific but the reason it seems most media has a sub is that there are millions of interested and capable people out there, so if a movie is only meaningful enough to only one person out of say 40 million, that's still one person who will make subs for that movie. We have the countless masses to thank, even when sometimes the quality isn't perfect.
8y ago

sidcool 8y ago

This is quite amazing. The code quality overall looks pretty good too. Pleasantly surprised to see it in Python. Was dreading complex C++ code. Thanks for sharing. The code will aid my learning.

joshstrange 8y ago

This looks pretty cool, right now I drop down to the command line to shift subs either direction but if they are off by too much or commercials/breaks don't line up then it's game over. I'll have to drop this out on my server, I'm tempted to run it across everything but I'll probably settle for running it manually when something doesn't line up in plex. Edit: Hmm, my first test moved the subtitles but they ended up 3-5 seconds after the audio instead of the original 2-5 before... I'll have to keep testing when I come across bad subs.

smacke 8y ago
My guess is that your subtitles don't have a "fixed drift". Unfortunately, this project only works for constant drift at the moment -- dynamic subtitle drift is a harder problem but absolutely worth thinking more about for future work.

hendry 8y ago

What about syncing up audio? Yet to find a CLI solution for syncing up & replacing external audio on my camera's MP4. Current solution is boot up FCPX and do a sync there which rips away minutes of my life.

gsich 8y ago
I use Audacity for this. I load both files, look for a peak or end of silence in both. The difference I remux with Mkvtoolnix. I do this with 2-3 spots. Loading the file is usually the longest waiting period.

bbx 8y ago

Thanks for this! I just happen to need it. I have the first season of my favourite US series, but dubbed in Spanish, because I want to train my ear. Unfortunately the subtitle files slowly drift apart from the audio by 10 seconds during each episode. I used to have a script to “stretch” subtitles manually, but it’s quite fiddly: https://gist.github.com/jgthms/7dfc20db3478a938069a0191c4e30843 https://gist.github.com/jgthms/7dfc20db3478a938069a0191c4e30...

smacke 8y ago
You are most welcome, although unfortunately it looks like this project doesn't support your use case yet. Right now it can only correct "constant drift", although I'll definitely be thinking more about other scenarios based on the reception. That said, it sounds like for your particular case, if you use VLC, you might be able to adjust the subtitle FPS so that you don't have to continuously manually resync. (I haven't run into this so just going off what others have mentioned.)

hantusk 8y ago

I've had great results with the Descript app which can both transcribe and can export a synchronized subtitle file. https://www.descript.com/ https://www.descript.com/ Perhaps you can import an srt file, and have it sync'ed? or perhaps you can export the srt text and let Descript sync it and re-create an .srt file.

aorth 8y ago

Super cool. This and the Python-based subtitle downloader subliminal make working with subtitles really easy. I had previously relied on incremental fudging of times using trial and error with www.sync-subtitles.com.

kowdermeister 8y ago

> I have yet to find a case where the automatic synchronization has been off by more than ~1 second. 1 second is in the 0 - 1000ms which is still a lot of room for manual tweaking in VLC. I can notice 50ms differences in sync.

EvagohP5 8y ago

I will love you for this. really. I can't tell how much time I have lost searching for the correct subtitles, in various languages

apexalpha 8y ago

Please tell me someone that works at Plex is reading this thread. I know it might seem a bit distant for Americans but this HUGE!

Guillaume86 8y ago
A Kodi plugin might be a more realistic target, subtitles handling in Kodi is pretty great in my experience, especially compared to Plex (download while playing, sources addons, offset adjustement).

boolemancer 8y ago

I recently ran across a similar tool for aligning subtitles based on a reference subtitle file called aligner: https://github.com/kaegi/aligner https://github.com/kaegi/aligner It seemed to work pretty well in my testing. Maybe it could be useful to look at?

smacke 8y ago
This project came up when I was researching this idea. It looks especially useful because the alignment algorithm is less restrictive (can introduce splits/breaks), so I plan on taking a closer look for sure. Given that it does not do any speech detection and requires a reference subtitles file, I'm not sure how well it would work for aligning the encoding for the speech-induced part, which will have a lot more noise. Particularly, the linked project works by aligning time intervals [a,b], while subsync is "dumber" in that it discretizes things into binary strings, and an interval constructed by merging consecutive 1's and 0's in speech detection-induced binary strings will probably be a lot harder to match to the subtitle intervals. Either way, I also thought this was a cool project and will be thinking more about whether any ideas can be borrowed.

s0adex 8y ago

It seems people are going back to P2P for movies. I'd forgotten the ordeals of unsynchronized srt files. Thanks to the Netflix era.

Krasnol 8y ago
Probably because not everything you want to see is on Netflix anymore and it's getting worse.

hobbes78 8y ago

I see this will only shift the timings some offset. But many times it isn't enough e.g. if the video source is NTSC and the subtitle were meant for PAL. In this scenario I use Subtitle Workshop, which allows me to specify two (or more) pairs of times, and all subtitles are synchronized accordingly. Optimally the two pairs of times would be near the beginning and the end, but to prevent big plot spoilers, I usually use the beginning and 2/3 of the length.