6 ms·

If I'm operating a cloud service like Netflix, then I'm already running thousands of ffmpeg processes on each machine. In other words, it's already a multi-core

If I'm operating a cloud service like Netflix, then I'm already running thousands of ffmpeg processes on each machine. In other words, it's already a multi-core job.

platzhirsch 3y ago
As multi-core as Python and Ruby then.
- bsdpufferfish 3y ago
  Yes. The kernel multiplies your efforts for you. It works great for web services.
sanitycheck 3y ago
Curious, what would that many ffmpeg processes be doing at Netflix? I assume new VOD content gets encoded once per format, and the amount of new content added per day is not gigantic. Agree with the general premise, of course, if I've got 10 different videos encoding at once then I don't need additional efficiency because the CPU's already maxed out.
- asveikau 3y ago
  Probably a lot more than once when you consider that different devices have different capabilities, and that they might stream you different bitrates depending on conditions like your network capability, screen resolution, how much you've paid them.. You could also imagine they might apply some kind of heuristic to decide to re-encode something based on some condition... Like fine tune encoder settings when a title becomes popular. No idea if they do that, just using some imagination.
- The_Colonel 3y ago
  I assume they re-compress for each resolution / format, quite possibly they also have different bitrate levels per resolution. Potentially even variants tweaked for certain classes of device (in cases this is not already covered by combination of format/resolution/bitrate). I would also assume they re-compress with new advances in video processing (things like HDR, improved compression). Also, their devs likely want fast feedback on changes - I imagine they might have CI running changes on some standard movies, checking various stats (like SNR) for regressions. Everybody loves if their CI finishes fast, so you might want to compress even a single movie in multiple threads.
  - sanitycheck 3y ago
    They'll be doing VBR encodes to DASH, HLS & (I guess still) MSS which covers the resolutions & formats... DRM will be what prevents high res content from working on some "less-trusted" platforms so the same encodes should work. (Plus a couple more "legacy" encodes with PIFF instead of CENC for ancient devices, probably.) New tech advances, sure, they probably do re-encode everything sometimes - even knocking a few MB off the size of a movie saves a measurable amount of $$ at that scale. But are there frequent enough tech advances to do that more than a couple of times a year..? The amount of difficult testing (every TV model group from the past 10 years, or something) required for an encode change is horrible. I'm sure they have better automation than anyone else, but I'm guessing it's still somewhat of a nightmare. Youtube, OTOH, I really can imagine having thousands of concurrent ffmpeg processes.
    - canucker2016 3y ago
      Why bring up assumptions/suppositions about Netflix's encoding process? Their tech blog and tech presentations discuss many of the requirements and steps involved for encoding source media to stream to all the devices that Netflix supports. The Netflix tech blog: https://netflixtechblog.com/ https://netflixtechblog.com/ or https://netflixtechblog.medium.com/ https://netflixtechblog.medium.com/ Netflix seems to use AWS CPU+GPU for encoding, whereas YouTube has gone to the expense of producing an ASIC to do much of their encoding. 2015 blog entry about their video encoding pipeline: https://netflixtechblog.com/high-quality-video-encoding-at-scale-d159db052746?gi=efeea4b4b5ac https://netflixtechblog.com/high-quality-video-encoding-at-s... 2021 presentation of their media encoding pipeline: https://www.infoq.com/presentations/video-encoding-netflix/ https://www.infoq.com/presentations/video-encoding-netflix/ An example of their FFmpeg usage - a neural-net video frame downscaler: https://netflixtechblog.com/for-your-eyes-only-improving-netflix-video-quality-with-neural-networks-5b8d032da09c https://netflixtechblog.com/for-your-eyes-only-improving-net... Their dynamic optimization encoding framework - allocating more bits for complex scenes and fewer bits for simpler, quieter scenes: https://netflixtechblog.com/dynamic-optimizer-a-perceptual-video-encoding-optimization-framework-e19f1e3a277f https://netflixtechblog.com/dynamic-optimizer-a-perceptual-v... and https://netflixtechblog.com/optimized-shot-based-encodes-now-streaming-4b9464204830 https://netflixtechblog.com/optimized-shot-based-encodes-now... Netflix developed an algorithm for determining video quality - VMAF, which helps determine their encoding decisions: https://netflixtechblog.com/toward-a-practical-perceptual-video-quality-metric-653f208b9652 https://netflixtechblog.com/toward-a-practical-perceptual-vi..., https://netflixtechblog.com/vmaf-the-journey-continues-44b51ee9ed12 https://netflixtechblog.com/vmaf-the-journey-continues-44b51..., https://netflixtechblog.com/toward-a-better-quality-metric-for-the-video-community-7ed94e752a30 https://netflixtechblog.com/toward-a-better-quality-metric-f...
- banana_giraffe 3y ago
  It's been reported in the past that Netflix encodes 120 different variants of each video they have [1] for different bitrates and different device's needs. And that was years ago, I wouldn't be surprised to learn it's a bigger number now. [1] https://news.ycombinator.com/item?id=4946275 https://news.ycombinator.com/item?id=4946275
The_Colonel 3y ago
I guess it's irrelevant for Netflix then*. But it sounds great for the remaining 99.99%. * I would be very surprised if Netflix even uses vanilla ffmpeg
- bsdpufferfish 3y ago
  > But it sounds great for the remaining 99.99%. I believe the vast majority of ffmpeg usages are web services, or one off encodings.
  - The_Colonel 3y ago
    Well, this feature is awesome for one-off encoding by a home user. Subjectively, me compressing my holiday video is much more important than Netflix re-compressing a million of them.
    - buu700 3y ago
      I use ffmpeg all the time, so this change is much appreciated. Well not really that often, but when I do encode video/audio it's generally with ffmpeg.
kevincox 3y ago
Latency is still valuable. For example YouTube (which IIRC uses ffmpeg) often takes hours to do transcodes. This is likely somewhat due to scheduling but assuming that they can get the same result doing 4x threads for 1/4 of the time they would prefer that as each job finishes faster. The only real question is at what efficiency cost the latency benefit stops being worth it.
- lelanthran 3y ago
  I think that if you're operating at the scale of Google using a single-threaded ffmpeg will finish your jobs in less time. If you have a queue of 100k videos to process and a cluster of 100 cores, assigning a video to each core as it becomes available is the most efficient way to process them, because your skipping the thread joining time. Anytime there is a queue of jobs, assigning the next job in the queue to the next free core is always going to be faster than assigning the next job to multiple cores.
- Thaxll 3y ago
  YouTube does not use ffmpeg, at the scale at which they operate it would be too slow / expensive. They use custom hardware just for encoding. fyi they have to transcode over 500h of videos per minute. So multiple that by all the formats they support. They operate at an insane scale, Netflix looks like a garage project for comparison.
  - astrange 3y ago
    There's still decoding. If a service claims to support all kinds of weird formats (like a MOV or AVI from the 90s) that means ffmpeg is running.
    - canucker2016 3y ago
      Google's use of ffmpeg: https://multimedia.cx/eggs/googles-youtube-uses-ffmpeg/ https://multimedia.cx/eggs/googles-youtube-uses-ffmpeg/ For encoding, recently, they've built their own ASIC to deal with H264 and VP9 encoding (for 7-33x faster encoding compared to CPU-only): https://arstechnica.com/gadgets/2021/04/youtube-is-now-building-its-own-video-transcoding-chips/ https://arstechnica.com/gadgets/2021/04/youtube-is-now-build...
stevehiehn 3y ago
Okaaay, and if I'm not operating a cloud service like Netflix, and I'm not running thousands of ffmpeg processes? In other words, it's not already a multi-core job?
- bsdpufferfish 3y ago
  This kind of multithreaded code introduces great complexity. So who wants to pay the cost for that tradeoff? Since most performance sensitive ffmpeg uses are cloud services, I don't see the benefit.

If I'm operating a cloud service like Netflix, then I'm already running thousands of ffmpeg processes on each machine. In other words, it's already a multi-core job.

platzhirsch 3y ago

As multi-core as Python and Ruby then.

bsdpufferfish 3y ago
Yes. The kernel multiplies your efforts for you. It works great for web services.

sanitycheck 3y ago

Curious, what would that many ffmpeg processes be doing at Netflix? I assume new VOD content gets encoded once per format, and the amount of new content added per day is not gigantic. Agree with the general premise, of course, if I've got 10 different videos encoding at once then I don't need additional efficiency because the CPU's already maxed out.

asveikau 3y ago
Probably a lot more than once when you consider that different devices have different capabilities, and that they might stream you different bitrates depending on conditions like your network capability, screen resolution, how much you've paid them.. You could also imagine they might apply some kind of heuristic to decide to re-encode something based on some condition... Like fine tune encoder settings when a title becomes popular. No idea if they do that, just using some imagination.
The_Colonel 3y ago
I assume they re-compress for each resolution / format, quite possibly they also have different bitrate levels per resolution. Potentially even variants tweaked for certain classes of device (in cases this is not already covered by combination of format/resolution/bitrate). I would also assume they re-compress with new advances in video processing (things like HDR, improved compression). Also, their devs likely want fast feedback on changes - I imagine they might have CI running changes on some standard movies, checking various stats (like SNR) for regressions. Everybody loves if their CI finishes fast, so you might want to compress even a single movie in multiple threads.
- sanitycheck 3y ago
  They'll be doing VBR encodes to DASH, HLS & (I guess still) MSS which covers the resolutions & formats... DRM will be what prevents high res content from working on some "less-trusted" platforms so the same encodes should work. (Plus a couple more "legacy" encodes with PIFF instead of CENC for ancient devices, probably.) New tech advances, sure, they probably do re-encode everything sometimes - even knocking a few MB off the size of a movie saves a measurable amount of $$ at that scale. But are there frequent enough tech advances to do that more than a couple of times a year..? The amount of difficult testing (every TV model group from the past 10 years, or something) required for an encode change is horrible. I'm sure they have better automation than anyone else, but I'm guessing it's still somewhat of a nightmare. Youtube, OTOH, I really can imagine having thousands of concurrent ffmpeg processes.
  - canucker2016 3y ago
    Why bring up assumptions/suppositions about Netflix's encoding process? Their tech blog and tech presentations discuss many of the requirements and steps involved for encoding source media to stream to all the devices that Netflix supports. The Netflix tech blog: https://netflixtechblog.com/ https://netflixtechblog.com/ or https://netflixtechblog.medium.com/ https://netflixtechblog.medium.com/ Netflix seems to use AWS CPU+GPU for encoding, whereas YouTube has gone to the expense of producing an ASIC to do much of their encoding. 2015 blog entry about their video encoding pipeline: https://netflixtechblog.com/high-quality-video-encoding-at-scale-d159db052746?gi=efeea4b4b5ac https://netflixtechblog.com/high-quality-video-encoding-at-s... 2021 presentation of their media encoding pipeline: https://www.infoq.com/presentations/video-encoding-netflix/ https://www.infoq.com/presentations/video-encoding-netflix/ An example of their FFmpeg usage - a neural-net video frame downscaler: https://netflixtechblog.com/for-your-eyes-only-improving-netflix-video-quality-with-neural-networks-5b8d032da09c https://netflixtechblog.com/for-your-eyes-only-improving-net... Their dynamic optimization encoding framework - allocating more bits for complex scenes and fewer bits for simpler, quieter scenes: https://netflixtechblog.com/dynamic-optimizer-a-perceptual-video-encoding-optimization-framework-e19f1e3a277f https://netflixtechblog.com/dynamic-optimizer-a-perceptual-v... and https://netflixtechblog.com/optimized-shot-based-encodes-now-streaming-4b9464204830 https://netflixtechblog.com/optimized-shot-based-encodes-now... Netflix developed an algorithm for determining video quality - VMAF, which helps determine their encoding decisions: https://netflixtechblog.com/toward-a-practical-perceptual-video-quality-metric-653f208b9652 https://netflixtechblog.com/toward-a-practical-perceptual-vi..., https://netflixtechblog.com/vmaf-the-journey-continues-44b51ee9ed12 https://netflixtechblog.com/vmaf-the-journey-continues-44b51..., https://netflixtechblog.com/toward-a-better-quality-metric-for-the-video-community-7ed94e752a30 https://netflixtechblog.com/toward-a-better-quality-metric-f...
banana_giraffe 3y ago
It's been reported in the past that Netflix encodes 120 different variants of each video they have [1] for different bitrates and different device's needs. And that was years ago, I wouldn't be surprised to learn it's a bigger number now. [1] https://news.ycombinator.com/item?id=4946275 https://news.ycombinator.com/item?id=4946275

The_Colonel 3y ago

I guess it's irrelevant for Netflix then*. But it sounds great for the remaining 99.99%. * I would be very surprised if Netflix even uses vanilla ffmpeg

bsdpufferfish 3y ago
> But it sounds great for the remaining 99.99%. I believe the vast majority of ffmpeg usages are web services, or one off encodings.
- The_Colonel 3y ago
  Well, this feature is awesome for one-off encoding by a home user. Subjectively, me compressing my holiday video is much more important than Netflix re-compressing a million of them.
  - buu700 3y ago
    I use ffmpeg all the time, so this change is much appreciated. Well not really that often, but when I do encode video/audio it's generally with ffmpeg.

kevincox 3y ago

Latency is still valuable. For example YouTube (which IIRC uses ffmpeg) often takes hours to do transcodes. This is likely somewhat due to scheduling but assuming that they can get the same result doing 4x threads for 1/4 of the time they would prefer that as each job finishes faster. The only real question is at what efficiency cost the latency benefit stops being worth it.

lelanthran 3y ago
I think that if you're operating at the scale of Google using a single-threaded ffmpeg will finish your jobs in less time. If you have a queue of 100k videos to process and a cluster of 100 cores, assigning a video to each core as it becomes available is the most efficient way to process them, because your skipping the thread joining time. Anytime there is a queue of jobs, assigning the next job in the queue to the next free core is always going to be faster than assigning the next job to multiple cores.
Thaxll 3y ago
YouTube does not use ffmpeg, at the scale at which they operate it would be too slow / expensive. They use custom hardware just for encoding. fyi they have to transcode over 500h of videos per minute. So multiple that by all the formats they support. They operate at an insane scale, Netflix looks like a garage project for comparison.
- astrange 3y ago
  There's still decoding. If a service claims to support all kinds of weird formats (like a MOV or AVI from the 90s) that means ffmpeg is running.
  - canucker2016 3y ago
    Google's use of ffmpeg: https://multimedia.cx/eggs/googles-youtube-uses-ffmpeg/ https://multimedia.cx/eggs/googles-youtube-uses-ffmpeg/ For encoding, recently, they've built their own ASIC to deal with H264 and VP9 encoding (for 7-33x faster encoding compared to CPU-only): https://arstechnica.com/gadgets/2021/04/youtube-is-now-building-its-own-video-transcoding-chips/ https://arstechnica.com/gadgets/2021/04/youtube-is-now-build...

stevehiehn 3y ago

Okaaay, and if I'm not operating a cloud service like Netflix, and I'm not running thousands of ffmpeg processes? In other words, it's not already a multi-core job?

bsdpufferfish 3y ago
This kind of multithreaded code introduces great complexity. So who wants to pay the cost for that tradeoff? Since most performance sensitive ffmpeg uses are cloud services, I don't see the benefit.