3 ms·

>why is a Packager necessary instead of virtually packaging and presenting an interface of a packaged file? Maybe I'm parsing your question wrong but perhaps t

by jasode 5y ago

>why is a Packager necessary instead of virtually packaging and presenting an interface of a packaged file? Maybe I'm parsing your question wrong but perhaps the confusion is cleared up if we remember that the "CDN" in the Netflix architecture diagram is not physically stored at AWS S3 : https://miro.medium.com/max/350/1*A5PR2QJ7STPbUd2xTg6z_g.png https://miro.medium.com/max/350/1*A5PR2QJ7STPbUd2xTg6z_g.png See: https://www.google.com/search?q=netflix+cdn+appliance+%22open+connect%22 https://www.google.com/search?q=netflix+cdn+appliance+%22ope... And a recent 2021 article says the Netflix CDN appliance has ~280 terabytes of local disk storage: https://dev.to/gbengelebs/netflix-system-design-how-netflix-onboards-new-content-2dlb https://dev.to/gbengelebs/netflix-system-design-how-netflix-... Thus, the "output" video files of Packager is eventually physically transferred to geographically distributed ISP datacenters. So to attempt to reconstruct Netflix's thought process... Once we work backwards from the need to eventually store copies of video content on distant Netflix CDN appliances, the question becomes which type of video file to store: (a) the codec-specific format (good for archival storage that can generate new downstream formats) -- but not optimized for client playback such as fast random seeking, a/v sync across dynamic changing resolutions, etc) -- or -- (b) the codec-agnostic format -- which is good for client device seeking, etc The option (a) wouldn't make sense since it you'd still need a cpu-intensive process (the "virtual packager to read chunks" in your words instead of The Packager) running at the ISP appliance to present client-device-optimized a/v streams. You'd have wasteful cpu cycles across multiple appliances creating the same "virtual package". So that leaves option (b) ... which means you need a batch process of some type (what Netflix calls Packager) running against AWS S3 storage to create a package for subsequent distribution to all appliances. Therefore, if your proposal of "virtually packaging and presenting an interface of a packaged file" can be reworded as "on-the-fly generated ephemeral a/v frames from the chunks" , their physical topology and need for efficient use of cpu wouldn't make that an optimal architecture.

dkarp 5y ago
Indeed, that's what I meant by: > Is it so that they can better distribute the packaged encodes to their CDN? The Packager, as far as I can tell, is not actually a cpu intensive process. That article mentions that the bottleneck was caused by IO which was resolved with the Virtual Assembler. It's not doing any encoding at all. It's stitching the encoded video together and packaging it with the audio/subtitles/metadata. It seems like this could be done at the edge without ever actually packaging the whole file. My guess is the same as yours though, the reason they're actually packaging the encodes is to send them to their CDN. But it's still a guess as maybe the actual reason is that the Virtual Package doesn't work for some other reason. It could just as well be to allow testing the packaged encode like you would any other deployment artifact before distributing.
- jasode 5y ago
  >The Packager, as far as I can tell, is not actually a cpu intensive process. [...] It's not doing any encoding at all. A Netflix employee can chime in but I assume cpu intensive since the desired objectives of Packager can only be achieved by transcoding which implies a cpu intensive process. E.g. Netflix wants to use AV1 as one the output formats for Packager. So any process that converts a codec-specific format to other coding-agnostic formats will require cpu. (e.g. Apple ProRes on the input files and AV1 on the output files.) Also more evidence that significant cpu is involved is this: >The overall ProRes video processing speed is increased from 50GB/Hour to 300GB/Hour. From a different perspective, the processing time to movie runtime ratio is reduced from 6:1 to about 1:1. [...] All the cloud packager instances now share a single scheduling queue with optimized compute resource utilization. The 300 gb/hr would be ~83 MB/sec which is way under the disk throughput S3 can provide. AWS says 100 Gbit/sec is possible which would be ~10 GB/sec : https://aws.amazon.com/premiumsupport/knowledge-center/s3-maximum-transfer-speed-ec2/ https://aws.amazon.com/premiumsupport/knowledge-center/s3-ma... >It seems like this could be done at the edge without ever actually packaging the whole file. But on-the-fly-ephemeral "virtual package" instead of a realized output disk file would lead to constant network traffic between the Netflix CDN appliance at the various ISP datacenters back to AWS S3. I guess this is technically possible but not sure what you gain with this alternative architecture instead of the CDN appliance just downloading one "package" file at off-peak hours (4:00am).
  - dkarp 5y ago
    > A Netflix employee can chime in but I assume cpu intensive since the desired objectives of Packager can only be achieved by transcoding I think you're maybe confusing the Packager with the Encoder. Transcoding (encoding) happens before packaging and is distributed. That's what the "encoded chunks" I've been talking about are. There is a different package for each encode, this is in the third linked article from the OP: https://netflixtechblog.com/high-quality-video-encoding-at-scale-d159db052746 https://netflixtechblog.com/high-quality-video-encoding-at-s... The package part seems similar to what is done by MKVMerge (https://en.wikipedia.org/wiki/MKVToolNix https://en.wikipedia.org/wiki/MKVToolNix) + the stitching of the chunked video encodes. There's very little processing necessary compared to encoding. MKVMerge will give you an mkv file from a H.264 video, audio files and subtitles in milliseconds and it doesn't matter how big the files are as it's just a container. They use a different container, but it's the same idea. You wouldn't have to read from S3, you could just push your encoded chunks to your CDN instead and use the edge server to virtual package them.

by jasode 5y ago

dkarp 5y ago

Indeed, that's what I meant by: > Is it so that they can better distribute the packaged encodes to their CDN? The Packager, as far as I can tell, is not actually a cpu intensive process. That article mentions that the bottleneck was caused by IO which was resolved with the Virtual Assembler. It's not doing any encoding at all. It's stitching the encoded video together and packaging it with the audio/subtitles/metadata. It seems like this could be done at the edge without ever actually packaging the whole file. My guess is the same as yours though, the reason they're actually packaging the encodes is to send them to their CDN. But it's still a guess as maybe the actual reason is that the Virtual Package doesn't work for some other reason. It could just as well be to allow testing the packaged encode like you would any other deployment artifact before distributing.

jasode 5y ago
>The Packager, as far as I can tell, is not actually a cpu intensive process. [...] It's not doing any encoding at all. A Netflix employee can chime in but I assume cpu intensive since the desired objectives of Packager can only be achieved by transcoding which implies a cpu intensive process. E.g. Netflix wants to use AV1 as one the output formats for Packager. So any process that converts a codec-specific format to other coding-agnostic formats will require cpu. (e.g. Apple ProRes on the input files and AV1 on the output files.) Also more evidence that significant cpu is involved is this: >The overall ProRes video processing speed is increased from 50GB/Hour to 300GB/Hour. From a different perspective, the processing time to movie runtime ratio is reduced from 6:1 to about 1:1. [...] All the cloud packager instances now share a single scheduling queue with optimized compute resource utilization. The 300 gb/hr would be ~83 MB/sec which is way under the disk throughput S3 can provide. AWS says 100 Gbit/sec is possible which would be ~10 GB/sec : https://aws.amazon.com/premiumsupport/knowledge-center/s3-maximum-transfer-speed-ec2/ https://aws.amazon.com/premiumsupport/knowledge-center/s3-ma... >It seems like this could be done at the edge without ever actually packaging the whole file. But on-the-fly-ephemeral "virtual package" instead of a realized output disk file would lead to constant network traffic between the Netflix CDN appliance at the various ISP datacenters back to AWS S3. I guess this is technically possible but not sure what you gain with this alternative architecture instead of the CDN appliance just downloading one "package" file at off-peak hours (4:00am).
- dkarp 5y ago
  > A Netflix employee can chime in but I assume cpu intensive since the desired objectives of Packager can only be achieved by transcoding I think you're maybe confusing the Packager with the Encoder. Transcoding (encoding) happens before packaging and is distributed. That's what the "encoded chunks" I've been talking about are. There is a different package for each encode, this is in the third linked article from the OP: https://netflixtechblog.com/high-quality-video-encoding-at-scale-d159db052746 https://netflixtechblog.com/high-quality-video-encoding-at-s... The package part seems similar to what is done by MKVMerge (https://en.wikipedia.org/wiki/MKVToolNix https://en.wikipedia.org/wiki/MKVToolNix) + the stitching of the chunked video encodes. There's very little processing necessary compared to encoding. MKVMerge will give you an mkv file from a H.264 video, audio files and subtitles in milliseconds and it doesn't matter how big the files are as it's just a container. They use a different container, but it's the same idea. You wouldn't have to read from S3, you could just push your encoded chunks to your CDN instead and use the edge server to virtual package them.