10 ms·

Rebuilding Netflix's video processing pipeline with microservices

197 points by samaysharma 3y ago

Zetobal 3y ago
You know I read articles like that all the time but the user experience of all these app gets worse. The time to first video frame on Netflix is not great to say the least. The rich metadata seems also only to be used internally...
- bodantogat 3y ago
  My pet peeve with Netflix is that it occasionally forgets where I last paused my video. I don't often watch Netflix, so when I do, it's annoying that I can't resume from where I left off
  - 0x6c6f6c 3y ago
    This is Disney+ but nearly every time. Moved from the living room to bedroom and had to seek to get back to where we were just last night
    - ceejayoz 3y ago
      I kinda wonder if it's a tracking endpoint my Pihole and/or Firefox privacy blockers are messing with.
    - foobarian 3y ago
      This is Paramount+ as well pretty much every time, and, and! After every ad break, the video rewinds to the start of the block before the ad break. So I have to fast forward to the ad break marker. And people wonder why we pirate :-)
    - elwell 3y ago
      I bet it saves them some server $. Probably keep more data client-side and batching to server.
  - _puk 3y ago
    I've found actively pausing a video before exiting seems to improve the accuracy of the resume point. I'm presuming this is because it fires a beacon on pause, but only every X (10?, 30?) seconds on playback. Not had many issues with it completely forgetting where I am though (it does sometimes get confused as to which episode I watched last though).
  - Grazester 3y ago
    I watch Netflix regularly and never had this issue. I wonder if it is your client? I use an Nvidia Shield(Android TV).
    - bodantogat 3y ago
      Possibly. I use my iPad for netflix.
- deleted 3y ago
  [deleted]
- SebFender 3y ago
  There you go - 100% right you are.
- vergessenmir 3y ago
  It might not be, but who are we comparing them to? Secondly who out there is doing this scale as "cheaply" Disney is already finding out that a streaming platform isn't cheap to run and hard to do efficiently.
  - deleted 3y ago
    [deleted]
franze 3y ago
[flagged]
eterpstra 3y ago
Wait, who am I supposed to believe here?!? Prime Video tore down their micro services in favor of a monolith just last year! Which trillion dollar globocorp is my tiny, insignificant company supposed to emulate? https://thenewstack.io/return-of-the-monolith-amazon-dumps-microservices-for-video-monitoring/ https://thenewstack.io/return-of-the-monolith-amazon-dumps-m...
- redberryi 3y ago
  Be thankful. That madness is what feeding a lot of families. Yours maybe one of those.
  - eterpstra 3y ago
    Mine is definitely one of those.
- teitoklien 3y ago
  [flagged]
  - Infinitesimus 3y ago
    Ridiculed by whom? We've seen many competitors try to make a streaming service and beyond Apple, they all provide a laggy experience even in the menus. If you're going to emulate someone, it's not a bad idea to emulate who has the best results
    - mtlmtlmtlmtl 3y ago
      Even Apple TV is pretty sluggish on my LG TV. And it sometimes makes my TV crash so I have to reboot it. It's maybe not as bad as the steaming pile of garbage that is Sky-Showtime, but it's got a ways to go before it's comparable to Netflix on my TV. Amazon Prime is pretty terrible on my TV too.
      - dangus 3y ago
SebFender 3y ago
These are conversations just like talking about code editors and so on. No one is right or wrong. But if I had to choose - micro services hard harder to control and secure. And vi? not for me ;)
highspeedbus 3y ago
Besides the debate about microservices being good or bad, it's clear that netflix developers are very passionate about what they do. To me this seems to play a big role in the success of a software product.
- imjonse 3y ago
  They are good for Netflix, bad for 99% of the other companies/projects, at least initially until they clearly know what they will build.
zer00eyz 3y ago
Did any of this: Make my nextflix better? How about cheaper? Did it deliver better content? Is this the work product of 2000 engineers focused on delivering me the worst content in the best way possible? What exactly am I getting for my 12, 20 ... wait what the hell is netflix charging now for their garbage content... 1000? 2000?d engineers at netflix and this is the article we get, this is their flex? I am underwhelmed.
- throwaway894345 3y ago
  I don’t like Netflix’s content either but I’m pretty sure that’s not up to the engineering department.
- matt_heimer 3y ago
  This is a bit like complaining about AWS because you don't like the product selection on Amazon.
  - barbazoo 3y ago
    Is Netflix offering their services as a platform now?
    - matt_heimer 3y ago
      Not sure why that would matter, complaining about content in response to an article about infrastructure doesn't make any more or less sense if the infrastructure is available as a service. But I do consider some of what Netflix does to be a platform. Most of it is platform in the sense that some of their open source offerings are commonly adopted such as Spinnaker. But if you look at the adoption of microservices at Netflix a part of that includes Conductor, an open-source microservice orchestration engine. The Netflix developers that created Conductor left Netflix and formed a new company named Orkes to offer Conductor as a platform. So while its not operated by Netflix, the microservice efforts they've made have been turned into a service offering.
  - ActionHank 3y ago
cletus 3y ago
Story time: I worked at Facebook and had to work with someone who came from Netflix. He was one of those people who, when he went to a new company, simply tried to reinvent everything he came from with no care or consideration given to what's already there. FB very much does not use microservices. The closest is in infra but the www layer is very much a massive monolith, probably too massive but that's another story. They've done some excellent engineering to make the developer experience pretty good, like you can commit to www and have it just push to prod within a few hours automatically (unless someone breaks trunk, which happens). Anyway, this person tried to reinvent everything as microservices and it pretty much just confirmed every preconceived notion (and hatred) or microservices that I already had. You create a whole bunch of issues with orchestration, versioning and deployment that you otherwise don't have. That's fine if you gain a huge benefit but often you just don't get any benefit at all. You simply get way more headaches in trying to debug why things aren't working. One of the key assumptions built into FB code that was broken is RYW (read your write). FB uses an in-memory write-through grraph database. On any given www request any writes you make will be consistent when you read them within that request. Every part of FB assumes this is true. This isn't true as soon as you cross an RPC boundary... much like you will with any microservices. So this caused no end of problems and the person just wouldn't hear this when it was identified as an issue before anything was done. So th enet effect was 2 years spent on a migration that ultimately was cancelled. Don't be that guy. When you go into a code base, realize that things are the way they are for a reason. It might not be a good reason. But there'll be a reason. Breaking things for the sake of reinventing the world how you think it should've been done were you starting from zero is just going to be a giant waste of everybody's time. As for Netflix video processing, they're basically encoding several thousand videos and deploying those segments to a VPN. This is nothing compared to, say, the video encoding needed for FB (let alone Youtube). Also, Netflix video processing is offline. This... isn't a hard problem. Netflix does do some cool stuff like AI scene detection to optimize encoding. But microservices feels like complete overkill.
- belval 3y ago
  > He was one of those people who, when he went to a new company, simply tried to reinvent everything he came from with no care or consideration given to what's already there. You have to have a very bad case of god complex if you look at a codebase that serves >3B users and experiences very little downtime thinking "oh yeah I could completely rearchitect that thing to be better"...
VyseofArcadia 3y ago
Does anyone have any reading material on the reliability of systems that use microservices? I've had a bit of basic probability rattling around in the back of my head that makes me suspect microservices are in general less reliable. I'd be interested in seeing a real-world analysis. My thinking goes like this, with some simplifying assumptions. Let's say you have a monolith with 99% uptime that you rearchitect into 5 microservices, each with 99% uptime, and if any one of those services goes down your whole system is down. Let's also assume for the sake of simplicity that these microservices are completely independent, although they are almost assuredly not. From basic probability, 99% uptime means there is some chunk of time t for which P(monolith goes down) = 1%. But P(microservice system goes down) = P(service A down or service B down or...) = P(service A down) + P(service B down) + ... = 5% In reality P(microservice system goes down) < 5% because they aren't independent and the chunk of time in which service A can go down will overlap that of service B. But still, that means the upper bound of the whole system going down is higher than for a monolith. But microservices are pretty popular, and I'm sure someone has thought along these lines before. One potential rebuttal is that each microservice is in fact more reliable than the monolith, although from what I've seen in my career I am skeptical that's truly the case. Where's the hole in my reasoning? (Or maybe I'm right. That would be fine too.)
- dmattia 3y ago
  In general, one of the goals of microservices should be that if one of the five services goes down, the other four should be able to operate in some capacity still. In practice, this can make the math quite a bit messier, but I don't think it necessarily has been worse overall from my perspective. So instead of having your system be up or down 99% of the time in a monolith, you'll have it fully up 95% of the time (using your numbers), but of that 5% of downtime, 20% of the time one of your products will be running slowly, or 10% of the time some new feature you launched won't work for specific customers in some specific region, etc. At my company it makes things like SLA/SLO guarantees for "our services" pretty complicated in that it's hard to define what uptime truly means, but overall I think the five microservice approach, when done well, should have less than 1% of complete downtime, at the cost of more partial downtime
belval 3y ago
> Processing Ad creatives posed some new challenges: media formats of Ads are quite different from movie and TV mezzanines that the team was familiar with, and there was a new set of media processing requirements related to the business needs of Ads. Nice to see them rearchitecture their service around enshittification.
bjornlouser 3y ago
'Long release cycles: The joint deployment meant that there was increased fear of unintended production outages as debugging and rollback can be difficult for a deployment of this size. This drove the approach of the “release train”. Every two weeks, a “snapshot” of all modules was taken, and promoted to be a “release candidate”. This release candidate then went through exhaustive testing which attempted to cover as large a surface area as possible. This testing stage took about two weeks. Thus, depending on when the code change was merged, it could take anywhere between two and four weeks to reach production.' I guess I'm just old, but I prefer the delay with a couple of weeks of testing versus pushing to prod and having the customer test the code.
- atomicfiredoll 3y ago
  What's interesting to me about this is that some companies seem to struggle to get even one reliable deployment process in place. In this case they were able to actively select the right process for the right job, even if it isn't the one they're normally geared toward using. It's not necessarily anything Earth shattering, but it may be an issue at some smaller places with fewer resources.
- mannyv 3y ago
  Netflix is a company that works with media people. You either understand what that implies or you don't. And remember, this is the backend for video encoding. Issues in this aren't necessarily user visible. The big benefit to their velocity is responsiveness. Apparently the backend team understands their customer and the timelines that the customer wants, and adjusted appropriately. Just dealing with ads would have been problematic, because those tend to be straight 1080 or 4k with stereo. Nothing fancy, but I'll bet they didn't fit inside the chunk size they were expecting since ads are usually 30 seconds or less. And they don't need the dynamic encoding etc that normal titles do. I wonder how much benefit dynamic encoding brings in space reduction?
  - neilv 3y ago
    > Netflix is a company that works with media people. You either understand what that implies or you don't. It implies everyone is hopped up on drugs?
b33j0r 3y ago
It’s not full-circle, as much as industry maturity. More stories lately are about “why we went pack to monoliths and building with borland C++” Not long ago it was more likely “how microservices solved everything at our company, and why only morons disagree.” So are we moving towards or away from microservices? Both. We’re maturing to use the right tools for the system.
- chrishare 3y ago
  True, and yet people are inclined to think the other grass is greener.
- sirsinsalot 3y ago
  Using the ... right ... tool for the job? In software development? Surely not! I want my dogmatic clickbait and LinkedIn-style grandstanding thank you very much! How else will I stay on the hedonic treadmill of staying up-to-date with a new framework or architecture every 3 months?!
trunnell 3y ago
Amazing to read about the continued re-invention of this part of Netflix's internal systems. I worked with this team (and its predecessors) during my time at Netflix. They achieved several "holy grails" of video encoding: a perceptual quality metric (VMAF), optimal bitrate selection per 2 second video chunk, and then optimal video chunking to be scene based rather than a fixed 2 seconds. Doing any of that in a research lab would be a challenge, but pulling it off at Netflix's scale is epic. You might need some background on how adaptive video streaming works to fully grok this article. But this is also just a story about a massive refactoring of a large, critical system. How many companies have you worked for that aggressively pursued refactoring/re-engineering their central systems? At most other places, I've seen risk aversion, fear, and mismanagement conspire to kill innovation. Not so at Netflix.
- entropicdrifter 3y ago
  That sounds like an epic series of projects to have gotten to work on.
- devindotcom 3y ago
  I'm curious about grain encoding - did you work on that at all? My friend heard they were doing a grain extraction layer that was re-added client side. Feel free to contact devin@techcrunch if you have any interesting insight.
  - mochomocha 3y ago
    Yes, that's part of the AV1 specs actually. See https://norkin.org/pdf/DCC_2018_AV1_film_grain.pdf https://norkin.org/pdf/DCC_2018_AV1_film_grain.pdf Andrey (who works for Netflix) drove the effort. Chatted with him about it.
  - aidenn0 3y ago
    I used to do a poor-mans version of that with low-bitrate mpeg4 (ASF or divX) movie downloads back in the day; the resolution would be low, and the MS MP4 codec tended to blur as well (plus noise killed the quality so much that there was often a denoiser in the encode pipeline), so adding any noise to the playback significantly improved the look of the movie.
issafram 3y ago
It's pretty easy, just stream the results of ffmpeg. I could create a Netflix platform in one day. /s if anyone took this seriously
dmillar 3y ago
Would be interested to also know how they handle per-title audio. With stereo -> 5.1 -> 7.1 and the sides and wide layout variants, how do they think about this during the inspection and encoding process? Being completely naive to Netflix's source media, and assuming it comes in a variety for formats and media, it seems like there are decision to make there. Though audio obviously has a much lower bandwidth burden, one would think there could still be QoE gains (and bandwidth savings) by doing things that AV1 can do with different scenes in something like OPUS.
devit 3y ago
I can't understand the article. If you want to encode video, use ffmpeg. Netflix serves static movies, so encoding is going to be relatively rare and can probably be done on whatever computing resources are already available. Quality-wise, the ffmpeg/x264/x265 people probably are doing a good job already. If you want to serve video, serve it with HLS or similar from static files stored on a CDN with a bunch of bitrate profiles. Here the problem is more creating or finding the CDN that anything to do with video. Can't quite figure out what the purpose of all the stuff in the article could be (maybe to justify the jobs of the people doing the work?)
- phantomathkg 3y ago
  Disclaimer, not working for Netflix, but working in this area. Ultimately it will still likely to be ffmpeg or their internal fork of it? But they are talking about per-title/per-shot optimisation here and not apply a blanket quality profile for every single video. Netflix also produce their own series and allow them to optimise their encoding further. They also have a Video Validation Services and Video Quality Services that do automatic quality check of the encoded video as the post indicated. Is it complex, yes it is. But is it overly complex, maybe not?
- ddorian43 3y ago
  Thats what they do, but better. Example they have their own CDN and put the nodes in your ISP facility. Using per-title-encoding, allows you to have less renditions for a given video, having better cache hits. Another example is pre-fetching data to the CDN nodes(when a new episode or season of a popular show comes out). Its just extra optimizations that come up and are worth it at scale.
- ksec 3y ago
  >Can't quite figure out what the purpose of all the stuff in the article could be Netflix encoding is actually much more complex. Both per Scene and per TiTle encoding, optimal bitrate selection etc are a lot more than what ffmpeg offers if you do it manually. Arguably you are right though Netflix could ignore all of that and just brute force the problem with much higher cost in terms of encoding, storage and bandwidth. But i guess at their scale it make sense to do all these optimisation.
ksec 3y ago
I am much more looking forward to Serving Netflix Video Traffic at 1600Gb/s and Beyond. Hoping drewg123 will share something soon :)
jiggawatts 3y ago
Okay, now that that's done, can someone at Netflix please figure out how to use their multiple data centres worth of distributed clusters to serve more than 4 or 5 subtitle languages please? If anyone at Netflix would like some assistance, I've previously consulted in the areas of large-scale compression optimisation, and I'm sure we can get those 100KB text files down to under 20KB! I'll help build distributed Kubernetes buzzword-compliant architectures, if that helps anyone get internal promotions as a part of this pan-cultural effort of inclusivity.

197 points by samaysharma 3y ago

You know I read articles like that all the time but the user experience of all these app gets worse. The time to first video frame on Netflix is not great to say the least. The rich metadata seems also only to be used internally...

bodantogat 3y ago
My pet peeve with Netflix is that it occasionally forgets where I last paused my video. I don't often watch Netflix, so when I do, it's annoying that I can't resume from where I left off
- 0x6c6f6c 3y ago
  This is Disney+ but nearly every time. Moved from the living room to bedroom and had to seek to get back to where we were just last night
  - ceejayoz 3y ago
    I kinda wonder if it's a tracking endpoint my Pihole and/or Firefox privacy blockers are messing with.
  - foobarian 3y ago
    This is Paramount+ as well pretty much every time, and, and! After every ad break, the video rewinds to the start of the block before the ad break. So I have to fast forward to the ad break marker. And people wonder why we pirate :-)
  - elwell 3y ago
    I bet it saves them some server $. Probably keep more data client-side and batching to server.
- _puk 3y ago
  I've found actively pausing a video before exiting seems to improve the accuracy of the resume point. I'm presuming this is because it fires a beacon on pause, but only every X (10?, 30?) seconds on playback. Not had many issues with it completely forgetting where I am though (it does sometimes get confused as to which episode I watched last though).
- Grazester 3y ago
  I watch Netflix regularly and never had this issue. I wonder if it is your client? I use an Nvidia Shield(Android TV).
  - bodantogat 3y ago
    Possibly. I use my iPad for netflix.
deleted 3y ago
[deleted]
SebFender 3y ago
There you go - 100% right you are.
vergessenmir 3y ago
It might not be, but who are we comparing them to? Secondly who out there is doing this scale as "cheaply" Disney is already finding out that a streaming platform isn't cheap to run and hard to do efficiently.
- deleted 3y ago
  [deleted]

franze 3y ago

[flagged]

eterpstra 3y ago

Wait, who am I supposed to believe here?!? Prime Video tore down their micro services in favor of a monolith just last year! Which trillion dollar globocorp is my tiny, insignificant company supposed to emulate? https://thenewstack.io/return-of-the-monolith-amazon-dumps-microservices-for-video-monitoring/ https://thenewstack.io/return-of-the-monolith-amazon-dumps-m...

redberryi 3y ago
Be thankful. That madness is what feeding a lot of families. Yours maybe one of those.
- eterpstra 3y ago
  Mine is definitely one of those.
teitoklien 3y ago
[flagged]
- Infinitesimus 3y ago
  Ridiculed by whom? We've seen many competitors try to make a streaming service and beyond Apple, they all provide a laggy experience even in the menus. If you're going to emulate someone, it's not a bad idea to emulate who has the best results
  - mtlmtlmtlmtl 3y ago
    Even Apple TV is pretty sluggish on my LG TV. And it sometimes makes my TV crash so I have to reboot it. It's maybe not as bad as the steaming pile of garbage that is Sky-Showtime, but it's got a ways to go before it's comparable to Netflix on my TV. Amazon Prime is pretty terrible on my TV too.
    - dangus 3y ago

SebFender 3y ago

These are conversations just like talking about code editors and so on. No one is right or wrong. But if I had to choose - micro services hard harder to control and secure. And vi? not for me ;)

highspeedbus 3y ago

Besides the debate about microservices being good or bad, it's clear that netflix developers are very passionate about what they do. To me this seems to play a big role in the success of a software product.

imjonse 3y ago
They are good for Netflix, bad for 99% of the other companies/projects, at least initially until they clearly know what they will build.

zer00eyz 3y ago

Did any of this: Make my nextflix better? How about cheaper? Did it deliver better content? Is this the work product of 2000 engineers focused on delivering me the worst content in the best way possible? What exactly am I getting for my 12, 20 ... wait what the hell is netflix charging now for their garbage content... 1000? 2000?d engineers at netflix and this is the article we get, this is their flex? I am underwhelmed.

throwaway894345 3y ago
I don’t like Netflix’s content either but I’m pretty sure that’s not up to the engineering department.
matt_heimer 3y ago
This is a bit like complaining about AWS because you don't like the product selection on Amazon.
- barbazoo 3y ago
  Is Netflix offering their services as a platform now?
  - matt_heimer 3y ago
    Not sure why that would matter, complaining about content in response to an article about infrastructure doesn't make any more or less sense if the infrastructure is available as a service. But I do consider some of what Netflix does to be a platform. Most of it is platform in the sense that some of their open source offerings are commonly adopted such as Spinnaker. But if you look at the adoption of microservices at Netflix a part of that includes Conductor, an open-source microservice orchestration engine. The Netflix developers that created Conductor left Netflix and formed a new company named Orkes to offer Conductor as a platform. So while its not operated by Netflix, the microservice efforts they've made have been turned into a service offering.
- ActionHank 3y ago

cletus 3y ago

Story time: I worked at Facebook and had to work with someone who came from Netflix. He was one of those people who, when he went to a new company, simply tried to reinvent everything he came from with no care or consideration given to what's already there. FB very much does not use microservices. The closest is in infra but the www layer is very much a massive monolith, probably too massive but that's another story. They've done some excellent engineering to make the developer experience pretty good, like you can commit to www and have it just push to prod within a few hours automatically (unless someone breaks trunk, which happens). Anyway, this person tried to reinvent everything as microservices and it pretty much just confirmed every preconceived notion (and hatred) or microservices that I already had. You create a whole bunch of issues with orchestration, versioning and deployment that you otherwise don't have. That's fine if you gain a huge benefit but often you just don't get any benefit at all. You simply get way more headaches in trying to debug why things aren't working. One of the key assumptions built into FB code that was broken is RYW (read your write). FB uses an in-memory write-through grraph database. On any given www request any writes you make will be consistent when you read them within that request. Every part of FB assumes this is true. This isn't true as soon as you cross an RPC boundary... much like you will with any microservices. So this caused no end of problems and the person just wouldn't hear this when it was identified as an issue before anything was done. So th enet effect was 2 years spent on a migration that ultimately was cancelled. Don't be that guy. When you go into a code base, realize that things are the way they are for a reason. It might not be a good reason. But there'll be a reason. Breaking things for the sake of reinventing the world how you think it should've been done were you starting from zero is just going to be a giant waste of everybody's time. As for Netflix video processing, they're basically encoding several thousand videos and deploying those segments to a VPN. This is nothing compared to, say, the video encoding needed for FB (let alone Youtube). Also, Netflix video processing is offline. This... isn't a hard problem. Netflix does do some cool stuff like AI scene detection to optimize encoding. But microservices feels like complete overkill.

belval 3y ago
> He was one of those people who, when he went to a new company, simply tried to reinvent everything he came from with no care or consideration given to what's already there. You have to have a very bad case of god complex if you look at a codebase that serves >3B users and experiences very little downtime thinking "oh yeah I could completely rearchitect that thing to be better"...

VyseofArcadia 3y ago

Does anyone have any reading material on the reliability of systems that use microservices? I've had a bit of basic probability rattling around in the back of my head that makes me suspect microservices are in general less reliable. I'd be interested in seeing a real-world analysis. My thinking goes like this, with some simplifying assumptions. Let's say you have a monolith with 99% uptime that you rearchitect into 5 microservices, each with 99% uptime, and if any one of those services goes down your whole system is down. Let's also assume for the sake of simplicity that these microservices are completely independent, although they are almost assuredly not. From basic probability, 99% uptime means there is some chunk of time t for which P(monolith goes down) = 1%. But P(microservice system goes down) = P(service A down or service B down or...) = P(service A down) + P(service B down) + ... = 5% In reality P(microservice system goes down) < 5% because they aren't independent and the chunk of time in which service A can go down will overlap that of service B. But still, that means the upper bound of the whole system going down is higher than for a monolith. But microservices are pretty popular, and I'm sure someone has thought along these lines before. One potential rebuttal is that each microservice is in fact more reliable than the monolith, although from what I've seen in my career I am skeptical that's truly the case. Where's the hole in my reasoning? (Or maybe I'm right. That would be fine too.)

dmattia 3y ago
In general, one of the goals of microservices should be that if one of the five services goes down, the other four should be able to operate in some capacity still. In practice, this can make the math quite a bit messier, but I don't think it necessarily has been worse overall from my perspective. So instead of having your system be up or down 99% of the time in a monolith, you'll have it fully up 95% of the time (using your numbers), but of that 5% of downtime, 20% of the time one of your products will be running slowly, or 10% of the time some new feature you launched won't work for specific customers in some specific region, etc. At my company it makes things like SLA/SLO guarantees for "our services" pretty complicated in that it's hard to define what uptime truly means, but overall I think the five microservice approach, when done well, should have less than 1% of complete downtime, at the cost of more partial downtime

belval 3y ago

> Processing Ad creatives posed some new challenges: media formats of Ads are quite different from movie and TV mezzanines that the team was familiar with, and there was a new set of media processing requirements related to the business needs of Ads. Nice to see them rearchitecture their service around enshittification.

bjornlouser 3y ago

'Long release cycles: The joint deployment meant that there was increased fear of unintended production outages as debugging and rollback can be difficult for a deployment of this size. This drove the approach of the “release train”. Every two weeks, a “snapshot” of all modules was taken, and promoted to be a “release candidate”. This release candidate then went through exhaustive testing which attempted to cover as large a surface area as possible. This testing stage took about two weeks. Thus, depending on when the code change was merged, it could take anywhere between two and four weeks to reach production.' I guess I'm just old, but I prefer the delay with a couple of weeks of testing versus pushing to prod and having the customer test the code.

atomicfiredoll 3y ago
What's interesting to me about this is that some companies seem to struggle to get even one reliable deployment process in place. In this case they were able to actively select the right process for the right job, even if it isn't the one they're normally geared toward using. It's not necessarily anything Earth shattering, but it may be an issue at some smaller places with fewer resources.
mannyv 3y ago
Netflix is a company that works with media people. You either understand what that implies or you don't. And remember, this is the backend for video encoding. Issues in this aren't necessarily user visible. The big benefit to their velocity is responsiveness. Apparently the backend team understands their customer and the timelines that the customer wants, and adjusted appropriately. Just dealing with ads would have been problematic, because those tend to be straight 1080 or 4k with stereo. Nothing fancy, but I'll bet they didn't fit inside the chunk size they were expecting since ads are usually 30 seconds or less. And they don't need the dynamic encoding etc that normal titles do. I wonder how much benefit dynamic encoding brings in space reduction?
- neilv 3y ago
  > Netflix is a company that works with media people. You either understand what that implies or you don't. It implies everyone is hopped up on drugs?

b33j0r 3y ago

It’s not full-circle, as much as industry maturity. More stories lately are about “why we went pack to monoliths and building with borland C++” Not long ago it was more likely “how microservices solved everything at our company, and why only morons disagree.” So are we moving towards or away from microservices? Both. We’re maturing to use the right tools for the system.

chrishare 3y ago
True, and yet people are inclined to think the other grass is greener.
sirsinsalot 3y ago
Using the ... right ... tool for the job? In software development? Surely not! I want my dogmatic clickbait and LinkedIn-style grandstanding thank you very much! How else will I stay on the hedonic treadmill of staying up-to-date with a new framework or architecture every 3 months?!

trunnell 3y ago

Amazing to read about the continued re-invention of this part of Netflix's internal systems. I worked with this team (and its predecessors) during my time at Netflix. They achieved several "holy grails" of video encoding: a perceptual quality metric (VMAF), optimal bitrate selection per 2 second video chunk, and then optimal video chunking to be scene based rather than a fixed 2 seconds. Doing any of that in a research lab would be a challenge, but pulling it off at Netflix's scale is epic. You might need some background on how adaptive video streaming works to fully grok this article. But this is also just a story about a massive refactoring of a large, critical system. How many companies have you worked for that aggressively pursued refactoring/re-engineering their central systems? At most other places, I've seen risk aversion, fear, and mismanagement conspire to kill innovation. Not so at Netflix.

entropicdrifter 3y ago
That sounds like an epic series of projects to have gotten to work on.
devindotcom 3y ago
I'm curious about grain encoding - did you work on that at all? My friend heard they were doing a grain extraction layer that was re-added client side. Feel free to contact devin@techcrunch if you have any interesting insight.
- mochomocha 3y ago
  Yes, that's part of the AV1 specs actually. See https://norkin.org/pdf/DCC_2018_AV1_film_grain.pdf https://norkin.org/pdf/DCC_2018_AV1_film_grain.pdf Andrey (who works for Netflix) drove the effort. Chatted with him about it.
- aidenn0 3y ago
  I used to do a poor-mans version of that with low-bitrate mpeg4 (ASF or divX) movie downloads back in the day; the resolution would be low, and the MS MP4 codec tended to blur as well (plus noise killed the quality so much that there was often a denoiser in the encode pipeline), so adding any noise to the playback significantly improved the look of the movie.

issafram 3y ago

It's pretty easy, just stream the results of ffmpeg. I could create a Netflix platform in one day. /s if anyone took this seriously

dmillar 3y ago

Would be interested to also know how they handle per-title audio. With stereo -> 5.1 -> 7.1 and the sides and wide layout variants, how do they think about this during the inspection and encoding process? Being completely naive to Netflix's source media, and assuming it comes in a variety for formats and media, it seems like there are decision to make there. Though audio obviously has a much lower bandwidth burden, one would think there could still be QoE gains (and bandwidth savings) by doing things that AV1 can do with different scenes in something like OPUS.

devit 3y ago

I can't understand the article. If you want to encode video, use ffmpeg. Netflix serves static movies, so encoding is going to be relatively rare and can probably be done on whatever computing resources are already available. Quality-wise, the ffmpeg/x264/x265 people probably are doing a good job already. If you want to serve video, serve it with HLS or similar from static files stored on a CDN with a bunch of bitrate profiles. Here the problem is more creating or finding the CDN that anything to do with video. Can't quite figure out what the purpose of all the stuff in the article could be (maybe to justify the jobs of the people doing the work?)

phantomathkg 3y ago
Disclaimer, not working for Netflix, but working in this area. Ultimately it will still likely to be ffmpeg or their internal fork of it? But they are talking about per-title/per-shot optimisation here and not apply a blanket quality profile for every single video. Netflix also produce their own series and allow them to optimise their encoding further. They also have a Video Validation Services and Video Quality Services that do automatic quality check of the encoded video as the post indicated. Is it complex, yes it is. But is it overly complex, maybe not?
ddorian43 3y ago
Thats what they do, but better. Example they have their own CDN and put the nodes in your ISP facility. Using per-title-encoding, allows you to have less renditions for a given video, having better cache hits. Another example is pre-fetching data to the CDN nodes(when a new episode or season of a popular show comes out). Its just extra optimizations that come up and are worth it at scale.
ksec 3y ago
>Can't quite figure out what the purpose of all the stuff in the article could be Netflix encoding is actually much more complex. Both per Scene and per TiTle encoding, optimal bitrate selection etc are a lot more than what ffmpeg offers if you do it manually. Arguably you are right though Netflix could ignore all of that and just brute force the problem with much higher cost in terms of encoding, storage and bandwidth. But i guess at their scale it make sense to do all these optimisation.

ksec 3y ago

I am much more looking forward to Serving Netflix Video Traffic at 1600Gb/s and Beyond. Hoping drewg123 will share something soon :)

jiggawatts 3y ago

Okay, now that that's done, can someone at Netflix please figure out how to use their multiple data centres worth of distributed clusters to serve more than 4 or 5 subtitle languages please? If anyone at Netflix would like some assistance, I've previously consulted in the areas of large-scale compression optimisation, and I'm sure we can get those 100KB text files down to under 20KB! I'll help build distributed Kubernetes buzzword-compliant architectures, if that helps anyone get internal promotions as a part of this pan-cultural effort of inclusivity.