6 ms·

I assume they re-compress for each resolution / format, quite possibly they also have different bitrate levels per resolution. Potentially even variants tweaked

I assume they re-compress for each resolution / format, quite possibly they also have different bitrate levels per resolution. Potentially even variants tweaked for certain classes of device (in cases this is not already covered by combination of format/resolution/bitrate). I would also assume they re-compress with new advances in video processing (things like HDR, improved compression). Also, their devs likely want fast feedback on changes - I imagine they might have CI running changes on some standard movies, checking various stats (like SNR) for regressions. Everybody loves if their CI finishes fast, so you might want to compress even a single movie in multiple threads.

sanitycheck 3y ago
They'll be doing VBR encodes to DASH, HLS & (I guess still) MSS which covers the resolutions & formats... DRM will be what prevents high res content from working on some "less-trusted" platforms so the same encodes should work. (Plus a couple more "legacy" encodes with PIFF instead of CENC for ancient devices, probably.) New tech advances, sure, they probably do re-encode everything sometimes - even knocking a few MB off the size of a movie saves a measurable amount of $$ at that scale. But are there frequent enough tech advances to do that more than a couple of times a year..? The amount of difficult testing (every TV model group from the past 10 years, or something) required for an encode change is horrible. I'm sure they have better automation than anyone else, but I'm guessing it's still somewhat of a nightmare. Youtube, OTOH, I really can imagine having thousands of concurrent ffmpeg processes.
- canucker2016 3y ago
  Why bring up assumptions/suppositions about Netflix's encoding process? Their tech blog and tech presentations discuss many of the requirements and steps involved for encoding source media to stream to all the devices that Netflix supports. The Netflix tech blog: https://netflixtechblog.com/ https://netflixtechblog.com/ or https://netflixtechblog.medium.com/ https://netflixtechblog.medium.com/ Netflix seems to use AWS CPU+GPU for encoding, whereas YouTube has gone to the expense of producing an ASIC to do much of their encoding. 2015 blog entry about their video encoding pipeline: https://netflixtechblog.com/high-quality-video-encoding-at-scale-d159db052746?gi=efeea4b4b5ac https://netflixtechblog.com/high-quality-video-encoding-at-s... 2021 presentation of their media encoding pipeline: https://www.infoq.com/presentations/video-encoding-netflix/ https://www.infoq.com/presentations/video-encoding-netflix/ An example of their FFmpeg usage - a neural-net video frame downscaler: https://netflixtechblog.com/for-your-eyes-only-improving-netflix-video-quality-with-neural-networks-5b8d032da09c https://netflixtechblog.com/for-your-eyes-only-improving-net... Their dynamic optimization encoding framework - allocating more bits for complex scenes and fewer bits for simpler, quieter scenes: https://netflixtechblog.com/dynamic-optimizer-a-perceptual-video-encoding-optimization-framework-e19f1e3a277f https://netflixtechblog.com/dynamic-optimizer-a-perceptual-v... and https://netflixtechblog.com/optimized-shot-based-encodes-now-streaming-4b9464204830 https://netflixtechblog.com/optimized-shot-based-encodes-now... Netflix developed an algorithm for determining video quality - VMAF, which helps determine their encoding decisions: https://netflixtechblog.com/toward-a-practical-perceptual-video-quality-metric-653f208b9652 https://netflixtechblog.com/toward-a-practical-perceptual-vi..., https://netflixtechblog.com/vmaf-the-journey-continues-44b51ee9ed12 https://netflixtechblog.com/vmaf-the-journey-continues-44b51..., https://netflixtechblog.com/toward-a-better-quality-metric-for-the-video-community-7ed94e752a30 https://netflixtechblog.com/toward-a-better-quality-metric-f...
  - astrange 3y ago
    > Their dynamic optimization encoding framework - allocating more bits for complex scenes and fewer bits for simpler, quieter scenes: https://netflixtechblog.com/dynamic-optimizer-a-perceptual-v https://netflixtechblog.com/dynamic-optimizer-a-perceptual-v... and https://netflixtechblog.com/optimized-shot-based-encodes-now https://netflixtechblog.com/optimized-shot-based-encodes-now... This is overrated - of course that's how you do it, what else would you do? > Mean-squared-error (MSE), typically used for encoder decisions, is a number that doesn’t always correlate very nicely with human perception. Academics, the reference MPEG encoder, and old proprietary encoder vendors like On2 VP9 did make decisions this way because their customers didn't know what they wanted. But people who care about quality, i.e. anime and movie pirate college students with a lot of free time, didn't. It looks like they've run x264 in an unnatural mode to get an improvement here, because the default "constant ratefactor" and "psy-rd" always behaved like this.
    - slhck 3y ago
      > This is overrated - of course that's how you do it, what else would you do? That's not what has been done previously for adaptive streaming. I guess you are referring to what encoding modes like CRF do for an individual, entire file? Or where else has this kind of approach been shown before? In the early days of streaming you would've done constant bitrate for MPEG-TS, even adding zero bytes to pad "easy" scenes. Later you'd have selected 2-pass ABR with some VBV bitrate constraints to not mess up the decoding buffer. At the time, YouTube did something where they tried to predict the CRF they'd need to achieve a certain (average) bitrate target (can't find the reference anymore). With per-title encoding (which was also popularized by Netflix) you could change the target bitrates for an entire title based on a previous complexity analysis. It took quite some time for other players in the field to also hop on the per-title encoding train. Going to a per-scene/per-shot level is the novely here, and exhaustively finding the best possible combination of QP/resolution pairs for an entire encoding ladder that also optimizes subjective quality – and not just MSE.
      - astrange 3y ago
        > exhaustively finding the best possible combination of QP/resolution pairs for an entire encoding ladder that also optimizes subjective quality – and not just MSE. This is unnecessary if the encoder is well-written. It's like how some people used to run multipass encoders 3 or 4 times just in case the result got better. You only need one analysis pass to find the optimal quality at a bitrate.

sanitycheck 3y ago

They'll be doing VBR encodes to DASH, HLS & (I guess still) MSS which covers the resolutions & formats... DRM will be what prevents high res content from working on some "less-trusted" platforms so the same encodes should work. (Plus a couple more "legacy" encodes with PIFF instead of CENC for ancient devices, probably.) New tech advances, sure, they probably do re-encode everything sometimes - even knocking a few MB off the size of a movie saves a measurable amount of $$ at that scale. But are there frequent enough tech advances to do that more than a couple of times a year..? The amount of difficult testing (every TV model group from the past 10 years, or something) required for an encode change is horrible. I'm sure they have better automation than anyone else, but I'm guessing it's still somewhat of a nightmare. Youtube, OTOH, I really can imagine having thousands of concurrent ffmpeg processes.

canucker2016 3y ago
Why bring up assumptions/suppositions about Netflix's encoding process? Their tech blog and tech presentations discuss many of the requirements and steps involved for encoding source media to stream to all the devices that Netflix supports. The Netflix tech blog: https://netflixtechblog.com/ https://netflixtechblog.com/ or https://netflixtechblog.medium.com/ https://netflixtechblog.medium.com/ Netflix seems to use AWS CPU+GPU for encoding, whereas YouTube has gone to the expense of producing an ASIC to do much of their encoding. 2015 blog entry about their video encoding pipeline: https://netflixtechblog.com/high-quality-video-encoding-at-scale-d159db052746?gi=efeea4b4b5ac https://netflixtechblog.com/high-quality-video-encoding-at-s... 2021 presentation of their media encoding pipeline: https://www.infoq.com/presentations/video-encoding-netflix/ https://www.infoq.com/presentations/video-encoding-netflix/ An example of their FFmpeg usage - a neural-net video frame downscaler: https://netflixtechblog.com/for-your-eyes-only-improving-netflix-video-quality-with-neural-networks-5b8d032da09c https://netflixtechblog.com/for-your-eyes-only-improving-net... Their dynamic optimization encoding framework - allocating more bits for complex scenes and fewer bits for simpler, quieter scenes: https://netflixtechblog.com/dynamic-optimizer-a-perceptual-video-encoding-optimization-framework-e19f1e3a277f https://netflixtechblog.com/dynamic-optimizer-a-perceptual-v... and https://netflixtechblog.com/optimized-shot-based-encodes-now-streaming-4b9464204830 https://netflixtechblog.com/optimized-shot-based-encodes-now... Netflix developed an algorithm for determining video quality - VMAF, which helps determine their encoding decisions: https://netflixtechblog.com/toward-a-practical-perceptual-video-quality-metric-653f208b9652 https://netflixtechblog.com/toward-a-practical-perceptual-vi..., https://netflixtechblog.com/vmaf-the-journey-continues-44b51ee9ed12 https://netflixtechblog.com/vmaf-the-journey-continues-44b51..., https://netflixtechblog.com/toward-a-better-quality-metric-for-the-video-community-7ed94e752a30 https://netflixtechblog.com/toward-a-better-quality-metric-f...
- astrange 3y ago
  > Their dynamic optimization encoding framework - allocating more bits for complex scenes and fewer bits for simpler, quieter scenes: https://netflixtechblog.com/dynamic-optimizer-a-perceptual-v https://netflixtechblog.com/dynamic-optimizer-a-perceptual-v... and https://netflixtechblog.com/optimized-shot-based-encodes-now https://netflixtechblog.com/optimized-shot-based-encodes-now... This is overrated - of course that's how you do it, what else would you do? > Mean-squared-error (MSE), typically used for encoder decisions, is a number that doesn’t always correlate very nicely with human perception. Academics, the reference MPEG encoder, and old proprietary encoder vendors like On2 VP9 did make decisions this way because their customers didn't know what they wanted. But people who care about quality, i.e. anime and movie pirate college students with a lot of free time, didn't. It looks like they've run x264 in an unnatural mode to get an improvement here, because the default "constant ratefactor" and "psy-rd" always behaved like this.
  - slhck 3y ago
    > This is overrated - of course that's how you do it, what else would you do? That's not what has been done previously for adaptive streaming. I guess you are referring to what encoding modes like CRF do for an individual, entire file? Or where else has this kind of approach been shown before? In the early days of streaming you would've done constant bitrate for MPEG-TS, even adding zero bytes to pad "easy" scenes. Later you'd have selected 2-pass ABR with some VBV bitrate constraints to not mess up the decoding buffer. At the time, YouTube did something where they tried to predict the CRF they'd need to achieve a certain (average) bitrate target (can't find the reference anymore). With per-title encoding (which was also popularized by Netflix) you could change the target bitrates for an entire title based on a previous complexity analysis. It took quite some time for other players in the field to also hop on the per-title encoding train. Going to a per-scene/per-shot level is the novely here, and exhaustively finding the best possible combination of QP/resolution pairs for an entire encoding ladder that also optimizes subjective quality – and not just MSE.
    - astrange 3y ago
      > exhaustively finding the best possible combination of QP/resolution pairs for an entire encoding ladder that also optimizes subjective quality – and not just MSE. This is unnecessary if the encoder is well-written. It's like how some people used to run multipass encoders 3 or 4 times just in case the result got better. You only need one analysis pass to find the optimal quality at a bitrate.