7 ms·

Netflix Cloud Packaging in the Terabyte Era

110 points by mmcclure 5y ago

londons_explore 5y ago
Packaging a movie is something that only needs to be done once right? Like filming the shots, hiring the actors, and writing the script? So why does a few hours matter in a total process which is many months or even years?
- maverwa 5y ago
  From what I understand, this is using in the production itself as well, potentially for steps inbetween? Especially the"The Conclusion" reads like its not a one-time thing but happens through the whole process?
  - wongarsu 5y ago
    Yes, the intro mentions "cloud-based post-production editing and collaboration pipelines", and the conclusion is "This significantly improves the Studio high quality proxy generation efficiency". While the article avoids mentioning why they actually need this, it reads like they have some collaboration platform for post-production of their own productions, and that involves frequent exports into a streamable format.
- MontyCarloHall 5y ago
  I assume that not all encodings of every movie are stored in perpetuity, to save space. Thus, sometimes they need to be regenerated on-demand, which is why efficiency is important.
  - endisneigh 5y ago
    Why not though? Isn’t storage cheap compared to compute? Hell, compared to the cost to license the movie storage is pennies.
    - MontyCarloHall 5y ago
      I’m sure someone at Netflix did the cost/benefit analysis and found that for extremely infrequently accessed encodings of certain movies (weird bitrate/resolution combinations), it’s actually cheaper to generate them dynamically once in a blue moon than to store them in perpetuity. > Hell, compared to the cost to license the movie storage is pennies. Yup. I’d be very curious to see the breakdown in how much Netflix spends on compute versus content production/licensing. I’d be willing to bet compute is an order or two of magnitude cheaper than content. I guess all these little optimizations add up? Or maybe their engineering management can’t see the forest from the trees.
      - twistedpair 5y ago
        I suspect they spend much more on transit than compute.
- deleted 5y ago
  [deleted]
- amitport 5y ago
  Once or a few times. It still saves money. Certainly in the Netflix scale. You invest some in engineering and you will save constant percentage $$ in the Netflix scale. Many smaller distributors don't invest internally in this kind of things (and don't have the internal engineering to achieve this). They will buy of-the-shelves solutions though.
  - kevindong 5y ago
    If the initial cost is x engineer hours and each hour costs $y, that implies the cost of the project is `x * $y + maintenance costs [of this system over the simpler, original approach]` Is the increased speed worth `x * $y + maintenance costs`? Could it pay off? Potentially yes. Will it actually pay off? I don't know since there's far too many variables to wildly speculate on. But I hope Netflix did the accounting at some point along the way to find out. Not every improvement is worth building since the initial investment might grossly overshadow any potential gains (as a purely hypothetical situation, that's why you shouldn't spend $1 million up front to save $1/day for the next 30 years).
MrBuddyCasino 5y ago
OT: starting with covid, Netflix reduced bitrate to relieve stress on overloaded networks. Did they ever reverse this? Video still looks overly compressed to me, so much so that I find it annoying.
- Widdershin 5y ago
  As far as I can tell Covid-quality is still in full effect, to the point where I don’t watch movies on Netflix at all anymore. I often start one, remember the quality and then find an alternate source. I wonder if they’ve seen that it doesn’t impact churn as much as it saves them money, and they’re keeping it that way as long as possible.
  - MrBuddyCasino 5y ago
    > I wonder if they’ve seen that it doesn’t impact churn as much as it saves them money, and they’re keeping it that way as long as possible. Yes thats my suspicion too.
- Aissen 5y ago
  It should have come back. If not, it's probably a bug, but good luck trying to report that to Netflix.
- tinus_hn 5y ago
  They forced SD resolution to appease technophobes who were crying Netflix was wasting bandwidth on ‘unnecessary entertainment’ while they were having trouble with their ‘necessary video meetings’. Which of course was a limitation of bandwidth across the internet while Netflix mostly uses local bandwidth, they try to stream from machines on the ISP networks because it saves them a ton of money. After about a month this was reversed. In the meantime it’s possible they have switched compression standards or settings which you may have noticed. But the covid limitations should be long gone. https://openconnect.netflix.com/en_gb/ https://openconnect.netflix.com/en_gb/
wongarsu 5y ago
So basically Netflix made their own network file system with S3 as a backing storage? This feels a bit like reinventing the wheel, but I can understand how you get there if you start with S3 and then need more and more functionality of a normal network-mounted file system.
- mkr-hn 5y ago
  It reminds me of how video game companies sometimes create whole virtual file systems in gigabyte+ blobs to deal with the limits of OS file systems. They're probably less useful in 2021 with SSDs becoming the norm. With hard drives, the thousands of files a game uses would end up in little fragments all over the platters and lead to horrible latency and head thrashing.
  - dcm360 5y ago
    At least hard drives still have decent seek times. Load time optimization for consoles with optical drives was another league: a single seek across the disk could lead to waiting a second longer on the loading screen.
    - dragontamer 5y ago
      Crash Bandicoot solved this problem by writing the level data in a streaming fashion, so that the CD-ROM would never have to seek while playing through a level. Of course, the CD-ROM only spins in one direction however. Crash Bandicoot programmers solved this issue by making it impossible to run backwards in the game. The game was basically one dimensional on purpose: you can only run forward, and never backwards, to optimize the CD-ROM bandwidth. Still, the game was groundbreaking, the gameplay and characters fresh. Its an incredible technical achievement, to the point where most people didn't even realize that running backwards was a technical hurdle they felt like not solving.
      - beefjerkins 5y ago
        > Of course, the CD-ROM only spins in one direction however. Crash Bandicoot programmers solved this issue by making it impossible to run backwards in the game. I've been playing crash since I was a kid, and I never thought about why you couldn't run backwards. What a brilliant solution.
theronmad 5y ago
Somehow too small a fine for such a corporation against the background of the European Union being sued by the giants monopilists Apple and Google. I recently published my https://essaywritinghelp.pro/ https://essaywritinghelp.pro/ essay on the practice of law and reviewed these lawsuits.
dkarp 5y ago
It's not entirely clear to me what the job of the Packager is here as opposed to a Virtual Package. After chunk encoding and Virtual Assembly with an index file, are Netflix actually packaging the encoded video/audio/subtitles/metadata into a single file for each encode that is pushed to their CDN? If so, then why is the Packager even necessary? Why not go one step further and create a Virtual Package as well? Is it so that they can better distribute the packaged encodes to their CDN?
- jasode 5y ago
  >It's not entirely clear to me what the job of the Packager is here. In the thread article's 1st paragraph is a link to a previous article with more details of Packager: https://netflixtechblog.com/packaging-award-winning-shows-with-award-winning-technology-c1010594ba39 https://netflixtechblog.com/packaging-award-winning-shows-wi...
  - dkarp 5y ago
    Right, and I actually read that. Maybe I should rephrase my question. I'd like to know why is a Packager necessary instead of virtually packaging and presenting an interface of a packaged file? Then you'd be able to keep your encoded chunks as chunks instead of having to download them and then upload them again to MezzFs.
    - jasode 5y ago
      >why is a Packager necessary instead of virtually packaging and presenting an interface of a packaged file? Maybe I'm parsing your question wrong but perhaps the confusion is cleared up if we remember that the "CDN" in the Netflix architecture diagram is not physically stored at AWS S3 : https://miro.medium.com/max/350/1*A5PR2QJ7STPbUd2xTg6z_g.png https://miro.medium.com/max/350/1*A5PR2QJ7STPbUd2xTg6z_g.png See: https://www.google.com/search?q=netflix+cdn+appliance+%22open+connect%22 https://www.google.com/search?q=netflix+cdn+appliance+%22ope... And a recent 2021 article says the Netflix CDN appliance has ~280 terabytes of local disk storage: https://dev.to/gbengelebs/netflix-system-design-how-netflix-onboards-new-content-2dlb https://dev.to/gbengelebs/netflix-system-design-how-netflix-... Thus, the "output" video files of Packager is eventually physically transferred to geographically distributed ISP datacenters. So to attempt to reconstruct Netflix's thought process... Once we work backwards from the need to eventually store copies of video content on distant Netflix CDN appliances, the question becomes which type of video file to store: (a) the codec-specific format (good for archival storage that can generate new downstream formats) -- but not optimized for client playback such as fast random seeking, a/v sync across dynamic changing resolutions, etc) -- or -- (b) the codec-agnostic format -- which is good for client device seeking, etc The option (a) wouldn't make sense since it you'd still need a cpu-intensive process (the "virtual packager to read chunks" in your words instead of The Packager) running at the ISP appliance to present client-device-optimized a/v streams. You'd have wasteful cpu cycles across multiple appliances creating the same "virtual package". So that leaves option (b) ... which means you need a batch process of some type (what Netflix calls Packager) running against AWS S3 storage to create a package for subsequent distribution to all appliances. Therefore, if your proposal of "virtually packaging and presenting an interface of a packaged file" can be reworded as "on-the-fly generated ephemeral a/v frames from the chunks" , their physical topology and need for efficient use of cpu wouldn't make that an optimal architecture.
xibalba 5y ago
> As Netflix becomes a producer of award winning content, the studio and content creation needs are also pushing the envelope of technology advancements This is a truly bizarre motivation. Should not the needs of the customer be the motivator for advancing technology? Also, Netflix has been winning awards for original content since 2013: https://en.wikipedia.org/wiki/List_of_accolades_received_by_Netflix https://en.wikipedia.org/wiki/List_of_accolades_received_by_...
- kwertyoowiyop 5y ago
  The rest of that paragraph does provide an example to justify that statement. Beyond that, they probably want to generate some content at very high resolutions, frame rates, and color ranges.

110 points by mmcclure 5y ago

Packaging a movie is something that only needs to be done once right? Like filming the shots, hiring the actors, and writing the script? So why does a few hours matter in a total process which is many months or even years?

maverwa 5y ago
From what I understand, this is using in the production itself as well, potentially for steps inbetween? Especially the"The Conclusion" reads like its not a one-time thing but happens through the whole process?
- wongarsu 5y ago
  Yes, the intro mentions "cloud-based post-production editing and collaboration pipelines", and the conclusion is "This significantly improves the Studio high quality proxy generation efficiency". While the article avoids mentioning why they actually need this, it reads like they have some collaboration platform for post-production of their own productions, and that involves frequent exports into a streamable format.
MontyCarloHall 5y ago
I assume that not all encodings of every movie are stored in perpetuity, to save space. Thus, sometimes they need to be regenerated on-demand, which is why efficiency is important.
- endisneigh 5y ago
  Why not though? Isn’t storage cheap compared to compute? Hell, compared to the cost to license the movie storage is pennies.
  - MontyCarloHall 5y ago
    I’m sure someone at Netflix did the cost/benefit analysis and found that for extremely infrequently accessed encodings of certain movies (weird bitrate/resolution combinations), it’s actually cheaper to generate them dynamically once in a blue moon than to store them in perpetuity. > Hell, compared to the cost to license the movie storage is pennies. Yup. I’d be very curious to see the breakdown in how much Netflix spends on compute versus content production/licensing. I’d be willing to bet compute is an order or two of magnitude cheaper than content. I guess all these little optimizations add up? Or maybe their engineering management can’t see the forest from the trees.
    - twistedpair 5y ago
      I suspect they spend much more on transit than compute.
deleted 5y ago
[deleted]
amitport 5y ago
Once or a few times. It still saves money. Certainly in the Netflix scale. You invest some in engineering and you will save constant percentage $$ in the Netflix scale. Many smaller distributors don't invest internally in this kind of things (and don't have the internal engineering to achieve this). They will buy of-the-shelves solutions though.
- kevindong 5y ago
  If the initial cost is x engineer hours and each hour costs $y, that implies the cost of the project is `x * $y + maintenance costs [of this system over the simpler, original approach]` Is the increased speed worth `x * $y + maintenance costs`? Could it pay off? Potentially yes. Will it actually pay off? I don't know since there's far too many variables to wildly speculate on. But I hope Netflix did the accounting at some point along the way to find out. Not every improvement is worth building since the initial investment might grossly overshadow any potential gains (as a purely hypothetical situation, that's why you shouldn't spend $1 million up front to save $1/day for the next 30 years).

MrBuddyCasino 5y ago

OT: starting with covid, Netflix reduced bitrate to relieve stress on overloaded networks. Did they ever reverse this? Video still looks overly compressed to me, so much so that I find it annoying.

Widdershin 5y ago
As far as I can tell Covid-quality is still in full effect, to the point where I don’t watch movies on Netflix at all anymore. I often start one, remember the quality and then find an alternate source. I wonder if they’ve seen that it doesn’t impact churn as much as it saves them money, and they’re keeping it that way as long as possible.
- MrBuddyCasino 5y ago
  > I wonder if they’ve seen that it doesn’t impact churn as much as it saves them money, and they’re keeping it that way as long as possible. Yes thats my suspicion too.
Aissen 5y ago
It should have come back. If not, it's probably a bug, but good luck trying to report that to Netflix.
tinus_hn 5y ago
They forced SD resolution to appease technophobes who were crying Netflix was wasting bandwidth on ‘unnecessary entertainment’ while they were having trouble with their ‘necessary video meetings’. Which of course was a limitation of bandwidth across the internet while Netflix mostly uses local bandwidth, they try to stream from machines on the ISP networks because it saves them a ton of money. After about a month this was reversed. In the meantime it’s possible they have switched compression standards or settings which you may have noticed. But the covid limitations should be long gone. https://openconnect.netflix.com/en_gb/ https://openconnect.netflix.com/en_gb/

wongarsu 5y ago

So basically Netflix made their own network file system with S3 as a backing storage? This feels a bit like reinventing the wheel, but I can understand how you get there if you start with S3 and then need more and more functionality of a normal network-mounted file system.

mkr-hn 5y ago
It reminds me of how video game companies sometimes create whole virtual file systems in gigabyte+ blobs to deal with the limits of OS file systems. They're probably less useful in 2021 with SSDs becoming the norm. With hard drives, the thousands of files a game uses would end up in little fragments all over the platters and lead to horrible latency and head thrashing.
- dcm360 5y ago
  At least hard drives still have decent seek times. Load time optimization for consoles with optical drives was another league: a single seek across the disk could lead to waiting a second longer on the loading screen.
  - dragontamer 5y ago
    Crash Bandicoot solved this problem by writing the level data in a streaming fashion, so that the CD-ROM would never have to seek while playing through a level. Of course, the CD-ROM only spins in one direction however. Crash Bandicoot programmers solved this issue by making it impossible to run backwards in the game. The game was basically one dimensional on purpose: you can only run forward, and never backwards, to optimize the CD-ROM bandwidth. Still, the game was groundbreaking, the gameplay and characters fresh. Its an incredible technical achievement, to the point where most people didn't even realize that running backwards was a technical hurdle they felt like not solving.
    - beefjerkins 5y ago
      > Of course, the CD-ROM only spins in one direction however. Crash Bandicoot programmers solved this issue by making it impossible to run backwards in the game. I've been playing crash since I was a kid, and I never thought about why you couldn't run backwards. What a brilliant solution.

theronmad 5y ago

Somehow too small a fine for such a corporation against the background of the European Union being sued by the giants monopilists Apple and Google. I recently published my https://essaywritinghelp.pro/ https://essaywritinghelp.pro/ essay on the practice of law and reviewed these lawsuits.

dkarp 5y ago

It's not entirely clear to me what the job of the Packager is here as opposed to a Virtual Package. After chunk encoding and Virtual Assembly with an index file, are Netflix actually packaging the encoded video/audio/subtitles/metadata into a single file for each encode that is pushed to their CDN? If so, then why is the Packager even necessary? Why not go one step further and create a Virtual Package as well? Is it so that they can better distribute the packaged encodes to their CDN?

jasode 5y ago
>It's not entirely clear to me what the job of the Packager is here. In the thread article's 1st paragraph is a link to a previous article with more details of Packager: https://netflixtechblog.com/packaging-award-winning-shows-with-award-winning-technology-c1010594ba39 https://netflixtechblog.com/packaging-award-winning-shows-wi...
- dkarp 5y ago
  Right, and I actually read that. Maybe I should rephrase my question. I'd like to know why is a Packager necessary instead of virtually packaging and presenting an interface of a packaged file? Then you'd be able to keep your encoded chunks as chunks instead of having to download them and then upload them again to MezzFs.
  - jasode 5y ago
    >why is a Packager necessary instead of virtually packaging and presenting an interface of a packaged file? Maybe I'm parsing your question wrong but perhaps the confusion is cleared up if we remember that the "CDN" in the Netflix architecture diagram is not physically stored at AWS S3 : https://miro.medium.com/max/350/1*A5PR2QJ7STPbUd2xTg6z_g.png https://miro.medium.com/max/350/1*A5PR2QJ7STPbUd2xTg6z_g.png See: https://www.google.com/search?q=netflix+cdn+appliance+%22open+connect%22 https://www.google.com/search?q=netflix+cdn+appliance+%22ope... And a recent 2021 article says the Netflix CDN appliance has ~280 terabytes of local disk storage: https://dev.to/gbengelebs/netflix-system-design-how-netflix-onboards-new-content-2dlb https://dev.to/gbengelebs/netflix-system-design-how-netflix-... Thus, the "output" video files of Packager is eventually physically transferred to geographically distributed ISP datacenters. So to attempt to reconstruct Netflix's thought process... Once we work backwards from the need to eventually store copies of video content on distant Netflix CDN appliances, the question becomes which type of video file to store: (a) the codec-specific format (good for archival storage that can generate new downstream formats) -- but not optimized for client playback such as fast random seeking, a/v sync across dynamic changing resolutions, etc) -- or -- (b) the codec-agnostic format -- which is good for client device seeking, etc The option (a) wouldn't make sense since it you'd still need a cpu-intensive process (the "virtual packager to read chunks" in your words instead of The Packager) running at the ISP appliance to present client-device-optimized a/v streams. You'd have wasteful cpu cycles across multiple appliances creating the same "virtual package". So that leaves option (b) ... which means you need a batch process of some type (what Netflix calls Packager) running against AWS S3 storage to create a package for subsequent distribution to all appliances. Therefore, if your proposal of "virtually packaging and presenting an interface of a packaged file" can be reworded as "on-the-fly generated ephemeral a/v frames from the chunks" , their physical topology and need for efficient use of cpu wouldn't make that an optimal architecture.

xibalba 5y ago

> As Netflix becomes a producer of award winning content, the studio and content creation needs are also pushing the envelope of technology advancements This is a truly bizarre motivation. Should not the needs of the customer be the motivator for advancing technology? Also, Netflix has been winning awards for original content since 2013: https://en.wikipedia.org/wiki/List_of_accolades_received_by_Netflix https://en.wikipedia.org/wiki/List_of_accolades_received_by_...

kwertyoowiyop 5y ago
The rest of that paragraph does provide an example to justify that statement. Beyond that, they probably want to generate some content at very high resolutions, frame rates, and color ranges.