9 ms·

How Video Streaming Processing Works

306 points by dreampeppers99 5y ago

arkj 5y ago
This is a really good introduction to how video works. The only think I find missing from a production OTT environment is DRM. Also I assume when the author says "http" he means "https" there are very few (I knew only one and even they used https termination by Akamai) providers who use "http" these days for streaming even with DRM. If anyone is building an economically viable OTT platform they should consider building their own CDN. Insiders tell me the Singapore based OTT provider HOOQ went bankrupt paying huge bills to AWS and Akamai.
- capableweb 5y ago
  > Also I assume when the author says "http" he means "https" there are very few (I knew only one and even they used https termination by Akamai) providers who use "http" these days for streaming even with DRM. For what the guide is, I think HTTP is accurate enough, as HTTPS is not its own protocol, it's always HTTP+TLS/SSL, so just saying HTTP is good enough. The ones that need to add the extension, will know so. For video editors (and other parts of the target for this guide), it's more than likely superfluous information. Also, as far as I know, DRM and HTTPS have different use cases. The DRM we see in browsers are around controlling playback on the client device, not encrypting the content to/from the media server/client device, while HTTPS is the opposite. So if they are using DRM or not doesn't matter when we talk about using HTTPS or not.
mtrovo 5y ago
That's a really nice introduction. One newbie question: how is this influenced by DRM on the browser? Is it all the same plus some security on top or do videos with DRM use a proprietary codecs and packaging?
- manorwar8 5y ago
  there is a nice series explaining DRM here https://sander.saares.eu/post/drm-is-not-a-black-box-01/ https://sander.saares.eu/post/drm-is-not-a-black-box-01/ I think the DRM part is made during the packaging phase the https://howvideo.works/#packaging https://howvideo.works/#packaging (where they need to signalize the media has drm and also encrypt the media)
- GeneticGenesis 5y ago
  DRM'd video content uses the same video codecs and containers, but introduces segment encryption during the packaging phase. In most cases, this encryption is within the audio and video samples rather than on the entire segment. Most content is encrypted using MPEG Common Encryption (CENC) - though there are a couple of variants. Decryption keys are then exchanged using one of the common proprietary DRM protocols, usually Widevine (Google), Playready (Microsoft), or Fairplay (Apple). The CDM (Content Decryption Module) in the browser is then passed the decryption key, so the browser. Can decrypt the content for playback.
bambax 5y ago
Slightly OT, but the OP links to this video which I found very interesting: https://www.youtube.com/watch?v=VQOdmckqNro https://www.youtube.com/watch?v=VQOdmckqNro In it two guys discuss how to reconstruct an audio file from its image representation, and it turns out to be pretty straightforward. In the end they are discussing the legal implications of being able to reconstruct audio from an image: if you buy the rights to the image does it give you the right to the audio (probably not!) But what it makes me wonder is how one could maybe draw sound, or have some kind of generative art program that could be used to first draw a wave and then listen to it. Maybe this has been done already?
- dusted 5y ago
  not as an art project, but software like milkytracker allows you to directly draw (short) waveforms with the mouse and play them back to listen to
deleted 5y ago
[deleted]
pjc50 5y ago
Good top-level summary of an extremely complicated subject. The containers diagram is surprising since the arrow means "derived from" - the earlier formats are at the top, I initially thought the arrow was in the wrong direction. Containers are kind of a nightmare since there is a lot of hidden knowledge you need. Many of the tools will let you put any video in any container, but only certain combinations actually work properly across the various devices and operating systems you will actually need to use. It's easiest to stick to "H264 + AAC in an MPEG4 container". HLS is a horrible hack that also has the side effect of making it harder to download and store videos.
- GeneticGenesis 5y ago
  While HLS isn't the cleanest protocol (Yay for extensions of the M3U playlist format...), it's actually really good at what it's designed to do - provide reliable video streaming while using HTTP/S over variable networks. Ultimately, HLS isn't designed for downloading and storing videos, it's designed for streaming.
  - beardedetim 5y ago
    Came to say this exact thing. HLS and it's fancy brother LLHLS aren't storage formats like MP4/FLV are. I think of HLS as a playback format: I'd play a Playlist when watching a VOD/livestream but I'd probably save it as an MP4.
- dylan604 5y ago
  >making it harder to download and store videos. And all of the studios cheered! /s But in reality, it hasn't, right? There are so many tools available that will download a video as a single file for you that's not even an issue. You should try looking for the Wizard with that strawman.
  - ajb257 5y ago
    There's a difference between not being designed for something, and being designed to prevent something. HLS isn't designed to prevent saving, it's just optimised for streaming.
mrtksn 5y ago
Not to judge the content quality but I'm curious about the motivation to build this website. Why? Because in the recent days we were discussing the poor quality Google search results and one argument is that the results are poor because the quality of the content has degraded due to the motivations of the content creators, i.e. optimising everything for SEO and views, likes. etc. with guidance of analytics. So how this works? Is the creator of the website doing SEO here? Will this be sold later? Is it a project for a portfolio? Why are we getting this good quality content? Will it be surfaced by Google? Not that I suspect anything nefarious, just curious about how original content(beyond commentary on social media) is made these days. What made that content possible? Why the creator(s) spent time to create graphics and text and payed for a domain name and probably hosting?
- ziggus 5y ago
  Uh, if you read a bit about Leandro, you'll learn that he's a senior engineer at Grupo Globo in Brazil. I'll leave it to you to discover more about Globo.
- erikbye 5y ago
  Well, he does link to his personal blog, which contains a link to his LinkedIn. His motivation might be personal marketing, job hunting. Or, he just likes to share knowledge for free, some people still find joy in that, fortunately. Then there's that cliche of how you learn more (or cementing your knowledge) by teaching. Sometimes that's all the motivation I need.
- GeneticGenesis 5y ago
  We actually built How Video Works as a side project at Mux [1] (inspired by How DNS Works [2]) - there's a note about it at the top of the page. We have contributions from our own team as well as others in the industry. Our main motivation is to try to educate on the complexities and intracies of streaming video. Despite streaming video representing 80+% of the internet, it's all underpinned by a fairly small community of engineers, which we're eager to help grow through tools like this, and the Demuxed community [3]. Edit: I should also mention that Leandro was kind enough to adapt a this content from his amazing Digital Video Introduction [4] [1] https://mux.com/ https://mux.com/ [2] https://howdns.works/ https://howdns.works/ [3] https://2021.demuxed.com/ https://2021.demuxed.com/ [4] https://github.com/leandromoreira/digital_video_introduction https://github.com/leandromoreira/digital_video_introduction
choxi 5y ago
Does all the 1.7GB of the decoded video get copied to the GPU? Or is there some playback controller that knows how to read the “delta format” from the codecs and only copies deltas to the framebuffer? It still blows my mind that we can stream video at 60FPS. I was making an animation app that did frame-by-frame playback and 16.6ms goes by fast! Just unpacking a frame into memory and copying it to the GPU seemed like it took a while.
- pavlov 5y ago
  You shouldn’t copy the frame data to the GPU (assuming that’s literally what your code was doing). Instead create a GPU texture that’s backed by a fixed buffer in main memory. Decode into that buffer, unlock it, and draw using the texture. The GPU will do direct memory access over PCIe, avoiding the copy. The CPU can’t be writing into the buffer while the GPU may be reading from it, so you can either use locks or double buffering to synchronize access.
  - danachow 5y ago
    > Instead create a GPU texture that’s backed by a fixed buffer in main memory. Decode into that buffer, unlock it, and draw using the texture. The GPU will do direct memory access over PCIe, avoiding the copy. With a dedicated GPU with its own memory there still is usually a memory to memory copy, it just doesn’t have to involve the CPU.
    - pavlov 5y ago
      Yeah, like so many other things in the GPU world, main RAM texture storage is more of a hint to the graphics card driver — "this buffer isn't going away and won't change until I explicitly tell you otherwise". It definitely used to be that GPUs did real DMA texture reads though, at least in the early days of dedicated GPUs with fairly little local RAM. I'm thinking back to when the Mac OS X accelerated window compositor was introduced — the graphics RAM simply wouldn't have been enough to hold more than a handful of window buffers.
  - zoenolan 5y ago
rgovostes 5y ago
As good as thread as any to ask: I have a lot of footage that is recorded continuously over several hours. Most of the time nothing notable happens in the video, but I have a set of timestamps of interesting events. (Think security camera with motion detection events, though this is not that.) Currently I extract the frame at each interesting timestamp and serve them as JPEGs. I have a web-based playback UI that includes a scrubber and list of events to jump to a particular timestamp. I would love to upgrade to a real video player that streams the underlying video, allowing the user to replay footage surrounding a timestamp. I have to be able to reliably seek to an exact frame from the event list, though. I've been looking for something web-based, self-hostable, that doesn't require I transcode my media up front (e.g., break down into HLS chunks). I have few users accessing it at a time so I can transcode on the fly with some caching (though I think it is already H.262 or H.264). Is there anything suitable out there?
- dylan604 5y ago
  Just because a file is encoded as H.264 does not mean it is streamable. The encoding needs to be done in a way that makes streaming realistic. For the rest of it, I would suggest an ffmpeg solution on a server. You can have it re-encode just the requested times while encoding to a streaming friendly format. There are JS libraries available that allow you to use HLS in the native <video> tag.
- foo_barrio 5y ago
  I had a similar issue and worked completely around it by using ffmpeg to generate a video file around the point of interest. Instead of serving up the JPEG as in your case, I would serve a small 10s mp4. It was something like (-5s to +5s). Trying to directly stream the file and seek was unreliable for me and I didn't want to get into setting up a full streaming server.
- cogman10 5y ago
  What is possible is going to depend a lot on the CPU you have and the media you have. That said, ffmpeg is going to be the best tool (IMO) to handle this. You may also look at a tool like Vapoursynth or AviSynth if you want to do any sort of preprocessing to the images. If the video is H.262 (or it is H.264 at a insane bitrate like 50Mbps), I'd encourage transcoding to something not as bitrate heavy. AV1 and HEVC are 2 of the best in class targets (but require a LOT of computational horsepower... OK there is also technically VVC, but nothing really supports that). If time is of the essence, then I'd suggest looking into what sort of codecs are supported by your CPU/GPU. They won't give you great quality but they will give you very fast transcoding. You'll want to target the latest codec possible. H.264 is pretty old at this point, H.265 (HEVC) or vp9 will do a better job at a lower bitrate if your card supports either. They are also relatively well supported. VP9 is royalty free. If your GPU or CPU do not support any recent codec, you might look into the SVT encoders for AV1/VP9, and x264/5 for H.264 or H.265. All this said, if the codec is fine and at a streamable bitrate, ffmpeg totally supports copying the stream from timeslices. You'll have to play around with buffering some of the stream so you can have ffmpeg do the slicing, but it's not too hard. That's the best option if the stream is streamable (transcoding will always hurt quality). Oh, and you'll very likely want to compile ffmpeg from source. The version of ffmpeg bundled with your OS is (likely) really old and may not have the encoders you are after. It's a huge PITA, but worth it, IMO. Alternatively you can likely find a build with all the stuff you want... but you'll need a level of trust in the provider of that binary.
dceddia 5y ago
Good overview of all the parts involved! I was hoping they’d talk a little more about the timing aspects, and keeping audio and video in sync during playback. What I’ve learned from working on a video editor is that “keeping a/v in sync” is… sort of a misnomer? Or anyway, it sounds very “active”, like you’d have to line up all the frames and carefully set timers to play them or something. But in practice, the audio and video frames are interleaved in the file, and they naturally come out in order (ish - see replies). The audio plays at a known rate (like 44.1KHz) and every frame of audio and video has a “presentation timestamp”, and these timestamps (are supposed to) line up between the streams. So you’ve got the audio and video both coming out of the file at way-faster-than-realtime (ideally), and then the syncing ends up being more like: let the audio play, and hold back the next video frame until it’s time to show it. The audio updates a “clock” as it plays (with each audio frame’s timestamp), and a separate loop watches the clock until the next video frame’s time is up. There seems to be surprisingly little material out there on this stuff, but the most helpful I found was the “Build a video editor in 1000 lines” tutorial [0] along with this spinoff [1], in conjunction with a few hours spent poring over the ffplay.c code trying to figure out how it works. 0: http://dranger.com/ffmpeg/ http://dranger.com/ffmpeg/ 1: https://github.com/leandromoreira/ffmpeg-libav-tutorial https://github.com/leandromoreira/ffmpeg-libav-tutorial
- pjc50 5y ago
  > let the audio play, and hold back the next video frame until it’s time to show it. The audio updates a “clock” as it plays (with each audio frame’s timestamp), and a separate loop watches the clock until the next video frame’s time is up. Yes .. but. They're interleaved within the container, but the encoder does not guarantee that they will be properly interleaved or even that they will be particularly temporally close to each other. So if you're operating in "pull" mode, as you should, then you may find that in order to find the next video frame you need to de-container (even if you don't fully decode!) a bunch of audio frames that you don't need yet, or vice versa. The alternative is to operate in "push" mode: decode whatever frames come off the stream, audio or video, and push them into separate ring buffers for output. This is easier to write but tends to err on the side of buffering more than you need.
no_time 5y ago
What I'm curious about is the actual hardware behind video sharing sites. Like how can Streamable,Reddit,Twitter encode such a massive amount of videos at scale. Do they have GPU farms? Dedicated hardware encoding hardware that us mortals can't buy? I left out YT on purpose because they have practically endless money to throw at the problem.
- ggregoire 5y ago
  Just for reference, I've got a server with a single RTX 2080 that reencodes 32 HD streams in parallel with NVENC and the GPU is used at only 10%.
- VWWHFSfQ 5y ago
  the last I checked the video metadata streamable is just using ffmpeg
- GeneticGenesis 5y ago
  Great question! The real answer is it varies, but for H.264, most just encode on software right now, because GPUs are expensive (especially in the cloud), and the failure rates are really high (if you try to build your own). ffmpeg and lib264 is really fast on modern hardware with decent X86 extensions. It's also worth noting that YouTube also now builds its own transcoding chips [1], and AWS just launched a dedicated transcoding instances based on Xilinx chips: [1] https://arstechnica.com/gadgets/2021/04/youtube-is-now-building-its-own-video-transcoding-chips/ https://arstechnica.com/gadgets/2021/04/youtube-is-now-build... [2] https://aws.amazon.com/ec2/instance-types/vt1/ https://aws.amazon.com/ec2/instance-types/vt1/
- dmw_ng 5y ago
  Software can be done at utterly stupid multiples of realtime at SD resolutions with only a few cores, depending on your quality target. Cores are very cheap Fancy GPUs tend to support 8 or more HD streams, even consumer cards using patched drivers. Then you have dedicated accelerator hardware, these can pack a tremendous amount of transcode into a tiny package. For example on AWS you have vt1 instances which support 8 (or 16?) simultaneous full HD/SD/QHD ladders at 2x realtime for around $200/mo. In answer to your actual question, at least YouTube selectively transcodes using fancier/more specific methods according to the popularity of the content. They do the cheap thing in bulk and the high quality thing for the 1% of content folk actually watch
rlyshw 5y ago
A quick search for "latency" in here has one little hand-wavey blurb about Mux working to optimize HLS. >Using various content delivery networks, Mux is driving HTTP Live Streaming (HLS) latency down to the lowest levels possible levels, and partnering with the best services at every mile of delivery is crucial in supporting this continued goal. In my experience, HLS and even LLHLS are a nightmare for latency. I jokingly call it "High Latency Streaming", since it seems very hard to (reliably) obtain glass-to-glass latency in the LL range (under 4 seconds). Usually Latency with cloud streaming gets to at least 30+s. I've dabbled with implementing WebRTC solutions to obtain Ultra Low Latency (<1s) delivery but that is even more complicated and fragmented with all of the browsers vying for standardization. The solution I've cooked up in the lab with mediasoup requires an FFMPEG shim to convert from MPEGTS/h264 via UDP/SRT to MKV/YP9 via RTP, which of course drives up the latency. Mediasoup has a ton of opinionated quirks for RTP ingest too, of course. Still I've been able to prove out 400ms "glass-to-glass" which has been fun. I wonder if Mux or really anyone has intentions to deliver scalable, on cloud or on prem solutions to fill the web-native LL/Ultra LL void left by the death of flash. I'm aware of some niche solutions like Softvelum's nimble streamer, but I hate their business model and I don't know anything about their scalability.
- torginus 5y ago
  What's wrong with WebRTC? Other than it not being simple. In my experience it's supported well enough by browsers. On the hosting side, you've got Google's C++ implementation, or you there's a GStreamer backend, so you can hook it up with whatever GStreamer can output. In the stuff I'm doing for work, we can get well below 100ms latency out of it. Since Google uses it for Stadia, i'm pretty sure it can do far better than that? What do you need low latency for, what's your use case? Video conferencing? App/Game streaming?
  - slimscsi 5y ago
    Cost and scale. HTTP video is significantly cheaper to deliver because of the robust and competitive CDN market. You can deliver all your video via WebRTC with lower latency, but your bandwidth bill will be an order of magnitude higher.
cogman10 5y ago
A quibble I have with this is introducing video as capturing RGB. Digital video very rarely uses that color space. Instead, the YUV family of colorspaces is far more common. Further, the FPS is somewhat wonk. While computer games and some british TV will broadcast at 60fps, most video is instead at 20->30fps.
- datagram 5y ago
  On the contrary, I think Introducing video as RGB is definitely the right choice. RGB is familiar to most people, and is ultimately how things get displayed on your monitor. For digital video, YUV is really just a convenient format for compression reasons as far as I'm aware. I do think it would be neat to point out YUV in an aside though. The history behind the format's origins (compatibility with black-and-white TVs) is interesting. As far as the frame rate goes, I'm willing to bet that more people are aware of 60fps as a standard frame rate for videogames than 24fps as a standard frame rate for film.

arkj 5y ago

This is a really good introduction to how video works. The only think I find missing from a production OTT environment is DRM. Also I assume when the author says "http" he means "https" there are very few (I knew only one and even they used https termination by Akamai) providers who use "http" these days for streaming even with DRM. If anyone is building an economically viable OTT platform they should consider building their own CDN. Insiders tell me the Singapore based OTT provider HOOQ went bankrupt paying huge bills to AWS and Akamai.

capableweb 5y ago
> Also I assume when the author says "http" he means "https" there are very few (I knew only one and even they used https termination by Akamai) providers who use "http" these days for streaming even with DRM. For what the guide is, I think HTTP is accurate enough, as HTTPS is not its own protocol, it's always HTTP+TLS/SSL, so just saying HTTP is good enough. The ones that need to add the extension, will know so. For video editors (and other parts of the target for this guide), it's more than likely superfluous information. Also, as far as I know, DRM and HTTPS have different use cases. The DRM we see in browsers are around controlling playback on the client device, not encrypting the content to/from the media server/client device, while HTTPS is the opposite. So if they are using DRM or not doesn't matter when we talk about using HTTPS or not.

mtrovo 5y ago

That's a really nice introduction. One newbie question: how is this influenced by DRM on the browser? Is it all the same plus some security on top or do videos with DRM use a proprietary codecs and packaging?

manorwar8 5y ago
there is a nice series explaining DRM here https://sander.saares.eu/post/drm-is-not-a-black-box-01/ https://sander.saares.eu/post/drm-is-not-a-black-box-01/ I think the DRM part is made during the packaging phase the https://howvideo.works/#packaging https://howvideo.works/#packaging (where they need to signalize the media has drm and also encrypt the media)
GeneticGenesis 5y ago
DRM'd video content uses the same video codecs and containers, but introduces segment encryption during the packaging phase. In most cases, this encryption is within the audio and video samples rather than on the entire segment. Most content is encrypted using MPEG Common Encryption (CENC) - though there are a couple of variants. Decryption keys are then exchanged using one of the common proprietary DRM protocols, usually Widevine (Google), Playready (Microsoft), or Fairplay (Apple). The CDM (Content Decryption Module) in the browser is then passed the decryption key, so the browser. Can decrypt the content for playback.

bambax 5y ago

Slightly OT, but the OP links to this video which I found very interesting: https://www.youtube.com/watch?v=VQOdmckqNro https://www.youtube.com/watch?v=VQOdmckqNro In it two guys discuss how to reconstruct an audio file from its image representation, and it turns out to be pretty straightforward. In the end they are discussing the legal implications of being able to reconstruct audio from an image: if you buy the rights to the image does it give you the right to the audio (probably not!) But what it makes me wonder is how one could maybe draw sound, or have some kind of generative art program that could be used to first draw a wave and then listen to it. Maybe this has been done already?

dusted 5y ago
not as an art project, but software like milkytracker allows you to directly draw (short) waveforms with the mouse and play them back to listen to

deleted 5y ago

[deleted]

pjc50 5y ago

Good top-level summary of an extremely complicated subject. The containers diagram is surprising since the arrow means "derived from" - the earlier formats are at the top, I initially thought the arrow was in the wrong direction. Containers are kind of a nightmare since there is a lot of hidden knowledge you need. Many of the tools will let you put any video in any container, but only certain combinations actually work properly across the various devices and operating systems you will actually need to use. It's easiest to stick to "H264 + AAC in an MPEG4 container". HLS is a horrible hack that also has the side effect of making it harder to download and store videos.

GeneticGenesis 5y ago
While HLS isn't the cleanest protocol (Yay for extensions of the M3U playlist format...), it's actually really good at what it's designed to do - provide reliable video streaming while using HTTP/S over variable networks. Ultimately, HLS isn't designed for downloading and storing videos, it's designed for streaming.
- beardedetim 5y ago
  Came to say this exact thing. HLS and it's fancy brother LLHLS aren't storage formats like MP4/FLV are. I think of HLS as a playback format: I'd play a Playlist when watching a VOD/livestream but I'd probably save it as an MP4.
dylan604 5y ago
>making it harder to download and store videos. And all of the studios cheered! /s But in reality, it hasn't, right? There are so many tools available that will download a video as a single file for you that's not even an issue. You should try looking for the Wizard with that strawman.
- ajb257 5y ago
  There's a difference between not being designed for something, and being designed to prevent something. HLS isn't designed to prevent saving, it's just optimised for streaming.

mrtksn 5y ago

Not to judge the content quality but I'm curious about the motivation to build this website. Why? Because in the recent days we were discussing the poor quality Google search results and one argument is that the results are poor because the quality of the content has degraded due to the motivations of the content creators, i.e. optimising everything for SEO and views, likes. etc. with guidance of analytics. So how this works? Is the creator of the website doing SEO here? Will this be sold later? Is it a project for a portfolio? Why are we getting this good quality content? Will it be surfaced by Google? Not that I suspect anything nefarious, just curious about how original content(beyond commentary on social media) is made these days. What made that content possible? Why the creator(s) spent time to create graphics and text and payed for a domain name and probably hosting?

ziggus 5y ago
Uh, if you read a bit about Leandro, you'll learn that he's a senior engineer at Grupo Globo in Brazil. I'll leave it to you to discover more about Globo.
erikbye 5y ago
Well, he does link to his personal blog, which contains a link to his LinkedIn. His motivation might be personal marketing, job hunting. Or, he just likes to share knowledge for free, some people still find joy in that, fortunately. Then there's that cliche of how you learn more (or cementing your knowledge) by teaching. Sometimes that's all the motivation I need.
GeneticGenesis 5y ago
We actually built How Video Works as a side project at Mux [1] (inspired by How DNS Works [2]) - there's a note about it at the top of the page. We have contributions from our own team as well as others in the industry. Our main motivation is to try to educate on the complexities and intracies of streaming video. Despite streaming video representing 80+% of the internet, it's all underpinned by a fairly small community of engineers, which we're eager to help grow through tools like this, and the Demuxed community [3]. Edit: I should also mention that Leandro was kind enough to adapt a this content from his amazing Digital Video Introduction [4] [1] https://mux.com/ https://mux.com/ [2] https://howdns.works/ https://howdns.works/ [3] https://2021.demuxed.com/ https://2021.demuxed.com/ [4] https://github.com/leandromoreira/digital_video_introduction https://github.com/leandromoreira/digital_video_introduction

choxi 5y ago

Does all the 1.7GB of the decoded video get copied to the GPU? Or is there some playback controller that knows how to read the “delta format” from the codecs and only copies deltas to the framebuffer? It still blows my mind that we can stream video at 60FPS. I was making an animation app that did frame-by-frame playback and 16.6ms goes by fast! Just unpacking a frame into memory and copying it to the GPU seemed like it took a while.

pavlov 5y ago
You shouldn’t copy the frame data to the GPU (assuming that’s literally what your code was doing). Instead create a GPU texture that’s backed by a fixed buffer in main memory. Decode into that buffer, unlock it, and draw using the texture. The GPU will do direct memory access over PCIe, avoiding the copy. The CPU can’t be writing into the buffer while the GPU may be reading from it, so you can either use locks or double buffering to synchronize access.
- danachow 5y ago
  > Instead create a GPU texture that’s backed by a fixed buffer in main memory. Decode into that buffer, unlock it, and draw using the texture. The GPU will do direct memory access over PCIe, avoiding the copy. With a dedicated GPU with its own memory there still is usually a memory to memory copy, it just doesn’t have to involve the CPU.
  - pavlov 5y ago
    Yeah, like so many other things in the GPU world, main RAM texture storage is more of a hint to the graphics card driver — "this buffer isn't going away and won't change until I explicitly tell you otherwise". It definitely used to be that GPUs did real DMA texture reads though, at least in the early days of dedicated GPUs with fairly little local RAM. I'm thinking back to when the Mac OS X accelerated window compositor was introduced — the graphics RAM simply wouldn't have been enough to hold more than a handful of window buffers.
- zoenolan 5y ago

rgovostes 5y ago

As good as thread as any to ask: I have a lot of footage that is recorded continuously over several hours. Most of the time nothing notable happens in the video, but I have a set of timestamps of interesting events. (Think security camera with motion detection events, though this is not that.) Currently I extract the frame at each interesting timestamp and serve them as JPEGs. I have a web-based playback UI that includes a scrubber and list of events to jump to a particular timestamp. I would love to upgrade to a real video player that streams the underlying video, allowing the user to replay footage surrounding a timestamp. I have to be able to reliably seek to an exact frame from the event list, though. I've been looking for something web-based, self-hostable, that doesn't require I transcode my media up front (e.g., break down into HLS chunks). I have few users accessing it at a time so I can transcode on the fly with some caching (though I think it is already H.262 or H.264). Is there anything suitable out there?

dylan604 5y ago
Just because a file is encoded as H.264 does not mean it is streamable. The encoding needs to be done in a way that makes streaming realistic. For the rest of it, I would suggest an ffmpeg solution on a server. You can have it re-encode just the requested times while encoding to a streaming friendly format. There are JS libraries available that allow you to use HLS in the native <video> tag.
foo_barrio 5y ago
I had a similar issue and worked completely around it by using ffmpeg to generate a video file around the point of interest. Instead of serving up the JPEG as in your case, I would serve a small 10s mp4. It was something like (-5s to +5s). Trying to directly stream the file and seek was unreliable for me and I didn't want to get into setting up a full streaming server.
cogman10 5y ago
What is possible is going to depend a lot on the CPU you have and the media you have. That said, ffmpeg is going to be the best tool (IMO) to handle this. You may also look at a tool like Vapoursynth or AviSynth if you want to do any sort of preprocessing to the images. If the video is H.262 (or it is H.264 at a insane bitrate like 50Mbps), I'd encourage transcoding to something not as bitrate heavy. AV1 and HEVC are 2 of the best in class targets (but require a LOT of computational horsepower... OK there is also technically VVC, but nothing really supports that). If time is of the essence, then I'd suggest looking into what sort of codecs are supported by your CPU/GPU. They won't give you great quality but they will give you very fast transcoding. You'll want to target the latest codec possible. H.264 is pretty old at this point, H.265 (HEVC) or vp9 will do a better job at a lower bitrate if your card supports either. They are also relatively well supported. VP9 is royalty free. If your GPU or CPU do not support any recent codec, you might look into the SVT encoders for AV1/VP9, and x264/5 for H.264 or H.265. All this said, if the codec is fine and at a streamable bitrate, ffmpeg totally supports copying the stream from timeslices. You'll have to play around with buffering some of the stream so you can have ffmpeg do the slicing, but it's not too hard. That's the best option if the stream is streamable (transcoding will always hurt quality). Oh, and you'll very likely want to compile ffmpeg from source. The version of ffmpeg bundled with your OS is (likely) really old and may not have the encoders you are after. It's a huge PITA, but worth it, IMO. Alternatively you can likely find a build with all the stuff you want... but you'll need a level of trust in the provider of that binary.

dceddia 5y ago

Good overview of all the parts involved! I was hoping they’d talk a little more about the timing aspects, and keeping audio and video in sync during playback. What I’ve learned from working on a video editor is that “keeping a/v in sync” is… sort of a misnomer? Or anyway, it sounds very “active”, like you’d have to line up all the frames and carefully set timers to play them or something. But in practice, the audio and video frames are interleaved in the file, and they naturally come out in order (ish - see replies). The audio plays at a known rate (like 44.1KHz) and every frame of audio and video has a “presentation timestamp”, and these timestamps (are supposed to) line up between the streams. So you’ve got the audio and video both coming out of the file at way-faster-than-realtime (ideally), and then the syncing ends up being more like: let the audio play, and hold back the next video frame until it’s time to show it. The audio updates a “clock” as it plays (with each audio frame’s timestamp), and a separate loop watches the clock until the next video frame’s time is up. There seems to be surprisingly little material out there on this stuff, but the most helpful I found was the “Build a video editor in 1000 lines” tutorial [0] along with this spinoff [1], in conjunction with a few hours spent poring over the ffplay.c code trying to figure out how it works. 0: http://dranger.com/ffmpeg/ http://dranger.com/ffmpeg/ 1: https://github.com/leandromoreira/ffmpeg-libav-tutorial https://github.com/leandromoreira/ffmpeg-libav-tutorial

pjc50 5y ago
> let the audio play, and hold back the next video frame until it’s time to show it. The audio updates a “clock” as it plays (with each audio frame’s timestamp), and a separate loop watches the clock until the next video frame’s time is up. Yes .. but. They're interleaved within the container, but the encoder does not guarantee that they will be properly interleaved or even that they will be particularly temporally close to each other. So if you're operating in "pull" mode, as you should, then you may find that in order to find the next video frame you need to de-container (even if you don't fully decode!) a bunch of audio frames that you don't need yet, or vice versa. The alternative is to operate in "push" mode: decode whatever frames come off the stream, audio or video, and push them into separate ring buffers for output. This is easier to write but tends to err on the side of buffering more than you need.

no_time 5y ago

What I'm curious about is the actual hardware behind video sharing sites. Like how can Streamable,Reddit,Twitter encode such a massive amount of videos at scale. Do they have GPU farms? Dedicated hardware encoding hardware that us mortals can't buy? I left out YT on purpose because they have practically endless money to throw at the problem.

ggregoire 5y ago
Just for reference, I've got a server with a single RTX 2080 that reencodes 32 HD streams in parallel with NVENC and the GPU is used at only 10%.
VWWHFSfQ 5y ago
the last I checked the video metadata streamable is just using ffmpeg
GeneticGenesis 5y ago
Great question! The real answer is it varies, but for H.264, most just encode on software right now, because GPUs are expensive (especially in the cloud), and the failure rates are really high (if you try to build your own). ffmpeg and lib264 is really fast on modern hardware with decent X86 extensions. It's also worth noting that YouTube also now builds its own transcoding chips [1], and AWS just launched a dedicated transcoding instances based on Xilinx chips: [1] https://arstechnica.com/gadgets/2021/04/youtube-is-now-building-its-own-video-transcoding-chips/ https://arstechnica.com/gadgets/2021/04/youtube-is-now-build... [2] https://aws.amazon.com/ec2/instance-types/vt1/ https://aws.amazon.com/ec2/instance-types/vt1/
dmw_ng 5y ago
Software can be done at utterly stupid multiples of realtime at SD resolutions with only a few cores, depending on your quality target. Cores are very cheap Fancy GPUs tend to support 8 or more HD streams, even consumer cards using patched drivers. Then you have dedicated accelerator hardware, these can pack a tremendous amount of transcode into a tiny package. For example on AWS you have vt1 instances which support 8 (or 16?) simultaneous full HD/SD/QHD ladders at 2x realtime for around $200/mo. In answer to your actual question, at least YouTube selectively transcodes using fancier/more specific methods according to the popularity of the content. They do the cheap thing in bulk and the high quality thing for the 1% of content folk actually watch

rlyshw 5y ago

A quick search for "latency" in here has one little hand-wavey blurb about Mux working to optimize HLS. >Using various content delivery networks, Mux is driving HTTP Live Streaming (HLS) latency down to the lowest levels possible levels, and partnering with the best services at every mile of delivery is crucial in supporting this continued goal. In my experience, HLS and even LLHLS are a nightmare for latency. I jokingly call it "High Latency Streaming", since it seems very hard to (reliably) obtain glass-to-glass latency in the LL range (under 4 seconds). Usually Latency with cloud streaming gets to at least 30+s. I've dabbled with implementing WebRTC solutions to obtain Ultra Low Latency (<1s) delivery but that is even more complicated and fragmented with all of the browsers vying for standardization. The solution I've cooked up in the lab with mediasoup requires an FFMPEG shim to convert from MPEGTS/h264 via UDP/SRT to MKV/YP9 via RTP, which of course drives up the latency. Mediasoup has a ton of opinionated quirks for RTP ingest too, of course. Still I've been able to prove out 400ms "glass-to-glass" which has been fun. I wonder if Mux or really anyone has intentions to deliver scalable, on cloud or on prem solutions to fill the web-native LL/Ultra LL void left by the death of flash. I'm aware of some niche solutions like Softvelum's nimble streamer, but I hate their business model and I don't know anything about their scalability.

torginus 5y ago
What's wrong with WebRTC? Other than it not being simple. In my experience it's supported well enough by browsers. On the hosting side, you've got Google's C++ implementation, or you there's a GStreamer backend, so you can hook it up with whatever GStreamer can output. In the stuff I'm doing for work, we can get well below 100ms latency out of it. Since Google uses it for Stadia, i'm pretty sure it can do far better than that? What do you need low latency for, what's your use case? Video conferencing? App/Game streaming?
- slimscsi 5y ago
  Cost and scale. HTTP video is significantly cheaper to deliver because of the robust and competitive CDN market. You can deliver all your video via WebRTC with lower latency, but your bandwidth bill will be an order of magnitude higher.

cogman10 5y ago

A quibble I have with this is introducing video as capturing RGB. Digital video very rarely uses that color space. Instead, the YUV family of colorspaces is far more common. Further, the FPS is somewhat wonk. While computer games and some british TV will broadcast at 60fps, most video is instead at 20->30fps.

datagram 5y ago
On the contrary, I think Introducing video as RGB is definitely the right choice. RGB is familiar to most people, and is ultimately how things get displayed on your monitor. For digital video, YUV is really just a convenient format for compression reasons as far as I'm aware. I do think it would be neat to point out YUV in an aside though. The history behind the format's origins (compatibility with black-and-white TVs) is interesting. As far as the frame rate goes, I'm willing to bet that more people are aware of 60fps as a standard frame rate for videogames than 24fps as a standard frame rate for film.