5 ms·

Syncing audio and video timing for multicam is so difficult that it generally requires onsite highly paid camera operators to manage. I wonder what magic they

Syncing audio and video timing for multicam is so difficult that it generally requires onsite highly paid camera operators to manage. I wonder what magic they used (and what underlying data format for the video) to make this sync magic happen.

Ajedi32 2y ago
When you think about it's actually pretty ridiculous that this difficult in the first place, in the internet era. In principle all you really need is a standard way to tag each keyframe with an NTP-synced timestamp, and any camera with an internet connection would be able to achieve this effortlessly.
- deleted 2y ago
  [deleted]
- fwip 2y ago
  NTP time skew can be on the order of 100ms. If the audio/video is desynced by 30ms, it can definitely be noticeable. Some people are more sensitive than others, and it also depends on the particulars of the content. Edit: Another commenter points out that in some situations, it's even necessary to align the time that the frames start, which necessarily requires sub-frame precision.
  - Ajedi32 2y ago
    It's almost never going to be that bad unless you're going out over a terrible internet connection. Worst case you could have a way to configure one camera to run its own NTP server and have the others sync to it over LAN. That would get you sub-frame (<1ms) precision easily.
- solardev 2y ago
  Is NTP precise enough for this? I thought timestamps are usually in milliseconds, but audio sampling rates can be fractions of a millisecond? If you're off by a little, won't it create weird echoes or interference in the sound ?
  - dghlsakjg 2y ago
    Sound travels at 1 foot per millisecond roughly (standard atmospheric conditions). So a soundfeed with multiple microphones several feet apart already have this issue.
    - solardev 2y ago
      But that's actually part of the signal (useful information), isn't it, like for stereo or spatial recording setups? Presumably each performer or instrument's delay is a function of its distance to different microphones. vs NTP-introduced errors, which are just noise and not part of the intentional recording?
- adamomada 2y ago
  Oh c’mon there are people who have thought about it for more than the ten seconds you have and it’s not feasible and not effortless - you want to give every cam an internet gateway??
- ricktdotorg 2y ago
  this drove me insane back in ~2004/2005 as a longtime-NTP-loving sysadmin who was doing tech for an indie film company. they were ingesting tons of multicam MiniDV into FCP and their timecode discipline was NOT good. there was no simple answer to this back then, pure manual work. truly is kind of bonkers it has taken until ~2024 to solve this (kinda).
- milleramp 2y ago
  Modern cameras are using PTP for synchronization.
zhengyi13 2y ago
Dumb/ignorant question: I'm under impression that the "clack" used at the beginning of the stereotypical film take functions as a sound spike that multiple reels of film from different angles can be synchronized on - assuming I'm not way off base here, what about syncing audio/video on modern digital devices is more difficult?
- dylan604 2y ago
  It's a different type of sync than that. Syncing multicameras would ensure the scanning of each camera is aligned so that the start of a frame is at the same time on each camera. Traditionally, you needed cameras that had the ability to accept a sync reference signal provided from a sync generator by running a dedicated cable to each camera as well as all of the other video equipment like switchers, Chyron, VTRs, etc. In the bad old days, if you tried to switch between two cameras that were not in sync, you'd get a very unclean cut as the H/V sync between sources were not aligned. Today, switching equipment just "re-stripes" the timing from each input. The need for a house reference is not necessary with modern prosumer gear. For broadcast still supporting interlaced HD, they still run that reference cable.
- numpad0 2y ago
  Clapperboards are for syncing video and audio for a single camera. They don't always use the audio recorded by the video camera itself, so means to synchronize the two is/was needed.
dylan604 2y ago
Camera tech and syncing has become almost trivial. You can do shoots with a mix of cameras all feeding back to a central place without any kind of sync between them all without running any kind of sync.
gorkish 2y ago
It's not hard; it's just kept behind a very thin veil of 'pro' Timecode server and recorders that sink timecode are literally all you need. Everything has been standardized for multiple decades. Truthfully, anymore even this is starting to become unnecessary; most NLE's can just figure it out from the regular metadata and autocorrelation to sync the clocks. Just dump it in the timeline and everything lines up.
- geerlingguy 2y ago
  The clocks often drift if you have more than 5-10 minutes of continuous footage though, so TC is essential. Unfortunately you have to bump up to prosumer/pro tier cameras and recorders if you want TC inputs.
  - gorkish 2y ago
    Genlock is absolutely not needed to overcome clock skew; sync to nearest frame at cuts and resample the audio to the frame timings. Plus most modern equipment has really great clocks and I would not be surprised if it drifts less than hypothesized. Let's imagine we are shooting 24fps on a couple of iPhones; what is the clock error required to drift out by 1 frame over 10 minutes? The answer works out to about 70ppm. The iphone uses a MEMS oscillator with about a 5ppm error, so ... eh? Just doesn't seem likely to be a big issue for regular stuff. If you are shooting scientific data or something maybe this matters more, but you'll have to account for it in that context no matter what.
solardev 2y ago
Is that still the case these days? I filmed a multi cam performance a few weeks ago with an Android phone, a PC laptop, and an iPad. The software (Premiere and others) was able to automatically sync them up just by analyzing their waveforms and lining them up. (You just drag the clips into a timeline and then right-click to sync them... it takes a few seconds to process but it's all automatic) It was good enough that the audio lined up perfectly except in cases where a performer was standing next to one microphone and far from the other one, which created a noticeable echo delay in stereo. But I think that's just the speed of sound at work, not the software's fault. My understanding is that this is a normal feature in video apps these days, to the point that a lot of older standalone time sync apps are no longer for sale because it's all built-in now. I'm not a professional in this space though, just someone recording amateur videos for a music class.
nerdjon 2y ago
Is it really? I record multiple angles with 2 iPhones and a third track with a microphone and Final Cut Pro just syncs it all up via audio syncing (before this feature came out), and handles them not having the same start point just fine. I have never had an issue with things not syncing properly unless there was a recording issue with one of them. I imagine it could be a problem with video with no audio, but that seems like a trivial thing to fix with some cheap microphones attached to your camera. It seems like this uses the same audio sync after the fact to get everything in sync, one time I was testing and all I did was tap the microphone and didn't say anything. Of course the phones couldn't sync with no other audio. I got an alert it was unable to sync the clips.
btgeekboy 2y ago
I admit I have no idea how it's actually working, but Apple does have experience keeping audio in sync: AirPods. I suspect video's a similar process.
boffinAudio 2y ago
I wrote an app (MIKME) which does this - the apps purpose is to allow remote recording with a self-powered bluetooth microphone, so you can have "point-source" audio for your video projects - leave the microphone with the subject, but move the camera around.. It wasn't that difficult to get audio/video in sync .. I just used AVFoundation, which has plenty of functions for dealing with the sync issue. The biggest problem was getting the recordings off the microphones filesystem to the iOS device - we had to invent a new protocol for this ("lost and found protocol"), because Apple sure don't make it easy for devices to share data .. but once the files are copied, the sync is the easy part.

Ajedi32 2y ago

When you think about it's actually pretty ridiculous that this difficult in the first place, in the internet era. In principle all you really need is a standard way to tag each keyframe with an NTP-synced timestamp, and any camera with an internet connection would be able to achieve this effortlessly.

deleted 2y ago
[deleted]
fwip 2y ago
NTP time skew can be on the order of 100ms. If the audio/video is desynced by 30ms, it can definitely be noticeable. Some people are more sensitive than others, and it also depends on the particulars of the content. Edit: Another commenter points out that in some situations, it's even necessary to align the time that the frames start, which necessarily requires sub-frame precision.
- Ajedi32 2y ago
  It's almost never going to be that bad unless you're going out over a terrible internet connection. Worst case you could have a way to configure one camera to run its own NTP server and have the others sync to it over LAN. That would get you sub-frame (<1ms) precision easily.
solardev 2y ago
Is NTP precise enough for this? I thought timestamps are usually in milliseconds, but audio sampling rates can be fractions of a millisecond? If you're off by a little, won't it create weird echoes or interference in the sound ?
- dghlsakjg 2y ago
  Sound travels at 1 foot per millisecond roughly (standard atmospheric conditions). So a soundfeed with multiple microphones several feet apart already have this issue.
  - solardev 2y ago
    But that's actually part of the signal (useful information), isn't it, like for stereo or spatial recording setups? Presumably each performer or instrument's delay is a function of its distance to different microphones. vs NTP-introduced errors, which are just noise and not part of the intentional recording?
adamomada 2y ago
Oh c’mon there are people who have thought about it for more than the ten seconds you have and it’s not feasible and not effortless - you want to give every cam an internet gateway??
ricktdotorg 2y ago
this drove me insane back in ~2004/2005 as a longtime-NTP-loving sysadmin who was doing tech for an indie film company. they were ingesting tons of multicam MiniDV into FCP and their timecode discipline was NOT good. there was no simple answer to this back then, pure manual work. truly is kind of bonkers it has taken until ~2024 to solve this (kinda).
milleramp 2y ago
Modern cameras are using PTP for synchronization.

zhengyi13 2y ago

Dumb/ignorant question: I'm under impression that the "clack" used at the beginning of the stereotypical film take functions as a sound spike that multiple reels of film from different angles can be synchronized on - assuming I'm not way off base here, what about syncing audio/video on modern digital devices is more difficult?

dylan604 2y ago
It's a different type of sync than that. Syncing multicameras would ensure the scanning of each camera is aligned so that the start of a frame is at the same time on each camera. Traditionally, you needed cameras that had the ability to accept a sync reference signal provided from a sync generator by running a dedicated cable to each camera as well as all of the other video equipment like switchers, Chyron, VTRs, etc. In the bad old days, if you tried to switch between two cameras that were not in sync, you'd get a very unclean cut as the H/V sync between sources were not aligned. Today, switching equipment just "re-stripes" the timing from each input. The need for a house reference is not necessary with modern prosumer gear. For broadcast still supporting interlaced HD, they still run that reference cable.
numpad0 2y ago
Clapperboards are for syncing video and audio for a single camera. They don't always use the audio recorded by the video camera itself, so means to synchronize the two is/was needed.

dylan604 2y ago

Camera tech and syncing has become almost trivial. You can do shoots with a mix of cameras all feeding back to a central place without any kind of sync between them all without running any kind of sync.

gorkish 2y ago

It's not hard; it's just kept behind a very thin veil of 'pro' Timecode server and recorders that sink timecode are literally all you need. Everything has been standardized for multiple decades. Truthfully, anymore even this is starting to become unnecessary; most NLE's can just figure it out from the regular metadata and autocorrelation to sync the clocks. Just dump it in the timeline and everything lines up.

geerlingguy 2y ago
The clocks often drift if you have more than 5-10 minutes of continuous footage though, so TC is essential. Unfortunately you have to bump up to prosumer/pro tier cameras and recorders if you want TC inputs.
- gorkish 2y ago
  Genlock is absolutely not needed to overcome clock skew; sync to nearest frame at cuts and resample the audio to the frame timings. Plus most modern equipment has really great clocks and I would not be surprised if it drifts less than hypothesized. Let's imagine we are shooting 24fps on a couple of iPhones; what is the clock error required to drift out by 1 frame over 10 minutes? The answer works out to about 70ppm. The iphone uses a MEMS oscillator with about a 5ppm error, so ... eh? Just doesn't seem likely to be a big issue for regular stuff. If you are shooting scientific data or something maybe this matters more, but you'll have to account for it in that context no matter what.

solardev 2y ago

Is that still the case these days? I filmed a multi cam performance a few weeks ago with an Android phone, a PC laptop, and an iPad. The software (Premiere and others) was able to automatically sync them up just by analyzing their waveforms and lining them up. (You just drag the clips into a timeline and then right-click to sync them... it takes a few seconds to process but it's all automatic) It was good enough that the audio lined up perfectly except in cases where a performer was standing next to one microphone and far from the other one, which created a noticeable echo delay in stereo. But I think that's just the speed of sound at work, not the software's fault. My understanding is that this is a normal feature in video apps these days, to the point that a lot of older standalone time sync apps are no longer for sale because it's all built-in now. I'm not a professional in this space though, just someone recording amateur videos for a music class.

nerdjon 2y ago

Is it really? I record multiple angles with 2 iPhones and a third track with a microphone and Final Cut Pro just syncs it all up via audio syncing (before this feature came out), and handles them not having the same start point just fine. I have never had an issue with things not syncing properly unless there was a recording issue with one of them. I imagine it could be a problem with video with no audio, but that seems like a trivial thing to fix with some cheap microphones attached to your camera. It seems like this uses the same audio sync after the fact to get everything in sync, one time I was testing and all I did was tap the microphone and didn't say anything. Of course the phones couldn't sync with no other audio. I got an alert it was unable to sync the clips.

btgeekboy 2y ago

I admit I have no idea how it's actually working, but Apple does have experience keeping audio in sync: AirPods. I suspect video's a similar process.

boffinAudio 2y ago

I wrote an app (MIKME) which does this - the apps purpose is to allow remote recording with a self-powered bluetooth microphone, so you can have "point-source" audio for your video projects - leave the microphone with the subject, but move the camera around.. It wasn't that difficult to get audio/video in sync .. I just used AVFoundation, which has plenty of functions for dealing with the sync issue. The biggest problem was getting the recordings off the microphones filesystem to the iOS device - we had to invent a new protocol for this ("lost and found protocol"), because Apple sure don't make it easy for devices to share data .. but once the files are copied, the sync is the easy part.