4 ms·

How Video Works

241 points by circuit_ 6y ago

tester756 6y ago
I wish I knew all this earlier this year, so I'd be writting different thing as my thesis Great explaination tho.
Sean-Der 6y ago
I really love all the work the Mux team is doing! They don't just throw APIs over the wall. They are putting in lots of effort to educate/empower developers. This is good stuff, and not just propietary knowledge to sell something. Also check out the video-dev Slack[0] and demuxed. Pion WebRTC and WebRTC for the Curious was motivated by conversations I had with other developers in their Slack. [0] https://video-dev.herokuapp.com https://video-dev.herokuapp.com [1] https://demuxed.com https://demuxed.com
- mmcclure 6y ago
  Thank you so much for the kind words! For what it's worth, we're mods of video-dev, but we don't see that as "our" Slack, it's run by a bunch of folks in the online video community. Demuxed is similar in that we're organizers, but we 100% avoid it being a company conference. Also, WebRTC for the Curious was on HN a while back, but for those that didn't see it the first time around: https://webrtcforthecurious.com/ https://webrtcforthecurious.com/
AnotherGoodName 6y ago
One thing i'd love to see added here is a blurb about the importance of the location of the video metadata in the file. Specifically that you need to have the metadata at the start of the file rather than the end of the file (which is the default) for low-latency playback on web. Explained: Freshly recorded MPEG (and almost all other container types) typically saves the header at the end of the file. This is the logical place to store header information after recording as you just append to the end of the file you've just written. It's the default. Unfortunately having the header at the end of the file is terrible for web playback. A user must download the entire video before playback starts. You absolutely need to re-encode the video with FAST-Start set. The header location is the number one mistake that i've seen a lot of website and developers make. If you find your website videos have a spinner that's seconds long before the video playback starts check the encoding. Specifically check that you've set fast start. I've seen companies who have a perfectly reasonable static site behind a CDN spend a fortune hosting their videos with a third party to fix the latency issues they were seeing. The expensive third party was ultimately fixing the issue because they re-encoded the videos with fast start set. The reality is their existing solution backed by a CDN would also have worked if they encoded the videos correctly.
- mmcclure 6y ago
  Yep, you make a really good point, thanks for the feedback. I think this bit would probably make sense to live in the upcoming "processing" section. If anyone's interested, this is called the moov atom, and when you use "web playback" presets in tools like Handbrake, that's typically what it does is move that header to the beginning of the file. The progressive playback issues for files not optimized has improved somewhat when the file's host supports range requests, since browsers have gotten smart enough to try and check the end of the file for that header via range requests. If you're interested in more along the codec/container line, one of my colleagues gave a talk on the internals of MP4 containers at Demuxed a few years ago[0] and another gave a Streaming Media keynote on the history of codecs and containers, particularly as they relate to online video[1]. [0] https://www.youtube.com/watch?v=iJAPTY3B7yE https://www.youtube.com/watch?v=iJAPTY3B7yE [1] https://www.youtube.com/watch?v=9Qo3WfsK4vc https://www.youtube.com/watch?v=9Qo3WfsK4vc
- j1elo 6y ago
  This is what the MP4 format calls the "moov atom". Writing metadata at the end, instead of periodically interleaving it inside the file, is not only useless for web playback, but also for any creation process (i.e., live recording) which intends to call itself robust. Imagine recording a 1-hour long video, only to have some unrelated issue abruptly stopping it all (e.g. the application crashes, or the video camera suddenly loses power), thus rendering an unplayable file because the damned metadata didn't get to be written at the end of the file... (luckily there are post-process methods that in some cases are able to restore such broken files, but still, a priori the file will be unplayable)
  - pavlov 6y ago
    > This is what the MP4 format calls the "moov atom". One of the few surviving legacies of Apple technology created during the Jobs interregnum. Like everything else not from NeXT, the modern Apple has killed QuickTime, but they can’t eliminate its concepts and wacky constants from the MPEG-4 standard.
  - galad87 6y ago
    And that's the reason fragmented mp4 were invented.
tobyhinloopen 6y ago
Thanks. I learned something. Maybe now I can fix our video playback on our website
shmerl 6y ago
Why bring HLS (Apple only) instead of DASH as an example?
- xrd 6y ago
  DASH is right there, underneath HLS. You didn't read far enough.
8K832d7tNmiQ 6y ago
For those who handles video streaming, which software do you use in production to segmented the video? I've already tinkered around with ffmpeg including throwing around any movflags I could find and it will never be able to stream it properly in chunks. My current working solution is using MP4Box with this command MP4Box -dash 1000 -rap -frag-rap test.mp4
_a1_ 6y ago
Oh, I thought it was about VHS video for some reason. I guess I'm old enough to rant that this should be named 'how video streaming works', not 'how video works' :/
Jackypot 6y ago
Perhaps someone here can answer a question I've had for a while: if you stream a video which is actually a static image (think a song on YouTube, the 'video' is just the album cover) is there any way to optimise that? Or must the server stream that same image constantly as though it was a regular video?
- knaik94 6y ago
  That's optimized based on the video codec itself where you keep frames with major changes but surrounding frames can be optimized by encoding where certain pixels move. That's essentially what the bitrate tells us, how much of the bits are new and how many are copied from a previous frame. This is an oversimplification, but with static images, you would need a low bitrate after the initial few frames because it doesn't change. And bitrate is a proxy to how much data is transmitted while streaming, it's bits per second. So a more concise answer, the codecs used for video by YouTube have the optimization you're thinking about built in. It would send a "pixel didn't change from last frame" and not need to send all the color information for all the pixels. YouTube does not really do variable frame rate, and it's messy for editing, but it's another optimization that is possible and could be useful for the type of video you're describing. The article describes DASH which you would need to send the full frame initially every time segment but per segment the previously described concept still applies. I don't believe YouTube uses DASH for anything outside live streams.
- rocgf 6y ago
  I do not know the answer, but I would absolutely shocked if it was streamed as a regular video. To the best of my knowledge, there are many, many optimizations which go into video. This is a pretty good high-level video I came across - https://www.youtube.com/watch?v=r6Rp-uo6HmI https://www.youtube.com/watch?v=r6Rp-uo6HmI
  - LocalH 6y ago
    I mean, technically, it would be "streamed as a regular video", it's just that "regular video encoding" is pretty efficient at encoding a single unchanging image nowadays.
rememberlenny 6y ago
Thank you to the Mux team for putting this together! I've been using Mux as the basis of milk.video's transcoding and video serving from the start, and it's been an absolute pleasure to work with. This site is an incredible, concise, and comprehensive resource for people trying to better understand how video works. I attended the demuxed conference this year, and was exposed for the first time into the nitty gritty of how video works behind the scenes. Huge props for the content not being a sales pitch, and truly being educational and informative.

241 points by circuit_ 6y ago

I wish I knew all this earlier this year, so I'd be writting different thing as my thesis Great explaination tho.

I really love all the work the Mux team is doing! They don't just throw APIs over the wall. They are putting in lots of effort to educate/empower developers. This is good stuff, and not just propietary knowledge to sell something. Also check out the video-dev Slack[0] and demuxed. Pion WebRTC and WebRTC for the Curious was motivated by conversations I had with other developers in their Slack. [0] https://video-dev.herokuapp.com https://video-dev.herokuapp.com [1] https://demuxed.com https://demuxed.com

mmcclure 6y ago
Thank you so much for the kind words! For what it's worth, we're mods of video-dev, but we don't see that as "our" Slack, it's run by a bunch of folks in the online video community. Demuxed is similar in that we're organizers, but we 100% avoid it being a company conference. Also, WebRTC for the Curious was on HN a while back, but for those that didn't see it the first time around: https://webrtcforthecurious.com/ https://webrtcforthecurious.com/

AnotherGoodName 6y ago

One thing i'd love to see added here is a blurb about the importance of the location of the video metadata in the file. Specifically that you need to have the metadata at the start of the file rather than the end of the file (which is the default) for low-latency playback on web. Explained: Freshly recorded MPEG (and almost all other container types) typically saves the header at the end of the file. This is the logical place to store header information after recording as you just append to the end of the file you've just written. It's the default. Unfortunately having the header at the end of the file is terrible for web playback. A user must download the entire video before playback starts. You absolutely need to re-encode the video with FAST-Start set. The header location is the number one mistake that i've seen a lot of website and developers make. If you find your website videos have a spinner that's seconds long before the video playback starts check the encoding. Specifically check that you've set fast start. I've seen companies who have a perfectly reasonable static site behind a CDN spend a fortune hosting their videos with a third party to fix the latency issues they were seeing. The expensive third party was ultimately fixing the issue because they re-encoded the videos with fast start set. The reality is their existing solution backed by a CDN would also have worked if they encoded the videos correctly.

mmcclure 6y ago
Yep, you make a really good point, thanks for the feedback. I think this bit would probably make sense to live in the upcoming "processing" section. If anyone's interested, this is called the moov atom, and when you use "web playback" presets in tools like Handbrake, that's typically what it does is move that header to the beginning of the file. The progressive playback issues for files not optimized has improved somewhat when the file's host supports range requests, since browsers have gotten smart enough to try and check the end of the file for that header via range requests. If you're interested in more along the codec/container line, one of my colleagues gave a talk on the internals of MP4 containers at Demuxed a few years ago[0] and another gave a Streaming Media keynote on the history of codecs and containers, particularly as they relate to online video[1]. [0] https://www.youtube.com/watch?v=iJAPTY3B7yE https://www.youtube.com/watch?v=iJAPTY3B7yE [1] https://www.youtube.com/watch?v=9Qo3WfsK4vc https://www.youtube.com/watch?v=9Qo3WfsK4vc
j1elo 6y ago
This is what the MP4 format calls the "moov atom". Writing metadata at the end, instead of periodically interleaving it inside the file, is not only useless for web playback, but also for any creation process (i.e., live recording) which intends to call itself robust. Imagine recording a 1-hour long video, only to have some unrelated issue abruptly stopping it all (e.g. the application crashes, or the video camera suddenly loses power), thus rendering an unplayable file because the damned metadata didn't get to be written at the end of the file... (luckily there are post-process methods that in some cases are able to restore such broken files, but still, a priori the file will be unplayable)
- pavlov 6y ago
  > This is what the MP4 format calls the "moov atom". One of the few surviving legacies of Apple technology created during the Jobs interregnum. Like everything else not from NeXT, the modern Apple has killed QuickTime, but they can’t eliminate its concepts and wacky constants from the MPEG-4 standard.
- galad87 6y ago
  And that's the reason fragmented mp4 were invented.

tobyhinloopen 6y ago

Thanks. I learned something. Maybe now I can fix our video playback on our website

shmerl 6y ago

Why bring HLS (Apple only) instead of DASH as an example?

xrd 6y ago
DASH is right there, underneath HLS. You didn't read far enough.

8K832d7tNmiQ 6y ago

For those who handles video streaming, which software do you use in production to segmented the video? I've already tinkered around with ffmpeg including throwing around any movflags I could find and it will never be able to stream it properly in chunks. My current working solution is using MP4Box with this command MP4Box -dash 1000 -rap -frag-rap test.mp4

_a1_ 6y ago

Oh, I thought it was about VHS video for some reason. I guess I'm old enough to rant that this should be named 'how video streaming works', not 'how video works' :/

Jackypot 6y ago

Perhaps someone here can answer a question I've had for a while: if you stream a video which is actually a static image (think a song on YouTube, the 'video' is just the album cover) is there any way to optimise that? Or must the server stream that same image constantly as though it was a regular video?

knaik94 6y ago
That's optimized based on the video codec itself where you keep frames with major changes but surrounding frames can be optimized by encoding where certain pixels move. That's essentially what the bitrate tells us, how much of the bits are new and how many are copied from a previous frame. This is an oversimplification, but with static images, you would need a low bitrate after the initial few frames because it doesn't change. And bitrate is a proxy to how much data is transmitted while streaming, it's bits per second. So a more concise answer, the codecs used for video by YouTube have the optimization you're thinking about built in. It would send a "pixel didn't change from last frame" and not need to send all the color information for all the pixels. YouTube does not really do variable frame rate, and it's messy for editing, but it's another optimization that is possible and could be useful for the type of video you're describing. The article describes DASH which you would need to send the full frame initially every time segment but per segment the previously described concept still applies. I don't believe YouTube uses DASH for anything outside live streams.
rocgf 6y ago
I do not know the answer, but I would absolutely shocked if it was streamed as a regular video. To the best of my knowledge, there are many, many optimizations which go into video. This is a pretty good high-level video I came across - https://www.youtube.com/watch?v=r6Rp-uo6HmI https://www.youtube.com/watch?v=r6Rp-uo6HmI
- LocalH 6y ago
  I mean, technically, it would be "streamed as a regular video", it's just that "regular video encoding" is pretty efficient at encoding a single unchanging image nowadays.

rememberlenny 6y ago

Thank you to the Mux team for putting this together! I've been using Mux as the basis of milk.video's transcoding and video serving from the start, and it's been an absolute pleasure to work with. This site is an incredible, concise, and comprehensive resource for people trying to better understand how video works. I attended the demuxed conference this year, and was exposed for the first time into the nitty gritty of how video works behind the scenes. Huge props for the content not being a sales pitch, and truly being educational and informative.