4 ms·

> A Netflix employee can chime in but I assume cpu intensive since the desired objectives of Packager can only be achieved by transcoding I think you're maybe

by dkarp 5y ago

> A Netflix employee can chime in but I assume cpu intensive since the desired objectives of Packager can only be achieved by transcoding I think you're maybe confusing the Packager with the Encoder. Transcoding (encoding) happens before packaging and is distributed. That's what the "encoded chunks" I've been talking about are. There is a different package for each encode, this is in the third linked article from the OP: https://netflixtechblog.com/high-quality-video-encoding-at-scale-d159db052746 https://netflixtechblog.com/high-quality-video-encoding-at-s... The package part seems similar to what is done by MKVMerge (https://en.wikipedia.org/wiki/MKVToolNix https://en.wikipedia.org/wiki/MKVToolNix) + the stitching of the chunked video encodes. There's very little processing necessary compared to encoding. MKVMerge will give you an mkv file from a H.264 video, audio files and subtitles in milliseconds and it doesn't matter how big the files are as it's just a container. They use a different container, but it's the same idea. You wouldn't have to read from S3, you could just push your encoded chunks to your CDN instead and use the edge server to virtual package them.

jasode 5y ago
>I think you're maybe confusing the Packager with the Encoder. Transcoding (encoding) happens before packaging The way I used "transcoding" was to refer to Netflix's Packager process of converting (in their words) "codec-specific" elementary stream format to "codec-agnostic" with extra frame metadata. I should have used a different word than "transcode" to encompass that (especially if the input and output files are the same "codec" but just different containers) ... but whatever the underlying process is, it implies (some) cpu constraints because they're only processing at 83 MB/sec throughput from SSD disks on S3. My laptop doing a simple "mux" type of operation with ffmpeg or MKVMerge can concatenate streams into another container greater than 400 MB/sec. >You wouldn't have to read from S3, you could just push your encoded chunks to your CDN instead and use the edge server to virtual package them. The Netflix blog says the Packager is scanning/analyzing the input file for exact frame start and stop times and storing that knowledge as extra metadata to enable future clients to randomly skip around the video. Just focusing on that one algorithm tells us it's not something we want to do repeatedly (and virtually) on all the edge CDN servers. (I'm reminded of analogous situation in mp3 vbr format that doesn't have exact time frame timestamps built-in for random seeking. Therefore, skipping to exactly 45m17s of a 60 minute mp3 takes a long time as the audio player "scans" the mp3 from the beginning to "count up" to 45m17s. One can build an "index" for fast mp3 random seek but that requires pre-processing the whole mp3 that's more cpu intensive than a simple mux operation.)

by dkarp 5y ago

jasode 5y ago

>I think you're maybe confusing the Packager with the Encoder. Transcoding (encoding) happens before packaging The way I used "transcoding" was to refer to Netflix's Packager process of converting (in their words) "codec-specific" elementary stream format to "codec-agnostic" with extra frame metadata. I should have used a different word than "transcode" to encompass that (especially if the input and output files are the same "codec" but just different containers) ... but whatever the underlying process is, it implies (some) cpu constraints because they're only processing at 83 MB/sec throughput from SSD disks on S3. My laptop doing a simple "mux" type of operation with ffmpeg or MKVMerge can concatenate streams into another container greater than 400 MB/sec. >You wouldn't have to read from S3, you could just push your encoded chunks to your CDN instead and use the edge server to virtual package them. The Netflix blog says the Packager is scanning/analyzing the input file for exact frame start and stop times and storing that knowledge as extra metadata to enable future clients to randomly skip around the video. Just focusing on that one algorithm tells us it's not something we want to do repeatedly (and virtually) on all the edge CDN servers. (I'm reminded of analogous situation in mp3 vbr format that doesn't have exact time frame timestamps built-in for random seeking. Therefore, skipping to exactly 45m17s of a 60 minute mp3 takes a long time as the audio player "scans" the mp3 from the beginning to "count up" to 45m17s. One can build an "index" for fast mp3 random seek but that requires pre-processing the whole mp3 that's more cpu intensive than a simple mux operation.)