8 ms·

Making Netflix's Data Infrastructure Cost-Effective

132 points by el_duderino 6y ago

jonplackett 6y ago
When I read something like this I realise just how immense of a lead tech companies are building vs 'normal' companies. There's just layer of expertise in how to streamline and organise themselves that anyone without a 1000 strong engineering team and huge engineering budget just are never going to think about.
- ramraj07 6y ago
  Ill argue that you don't need it for a small company - just don't overcomplicate your life by using technology that's designed for big companies. Stick to simple systems like elastic beanstalk or app engine + RDS and something like snowflake and you get powerful, manageable, auditable projects. I'm also finding great benefit in launching each project in its own AWS account. This means you know exactly what project costs how much!
  - jonplackett 6y ago
    I was thinking more in terms of large but traditional companies. The kind tech companies put out of business regularly!
- QUFB 6y ago
  This is where things are heading, especially for business or corporate 'consumers': Need web hosting? Certainly you can't deal with a 2Tbps DDoS attack, so let's route that traffic through Cloudflare. Sending email? Can't send it out with those /23s you have, they're on some blacklist from a shady DNSBL provider. Need to pay for GSuite or Office365. Can't protect your CEO against a targeted attack? GSuite Advanced Protection or Microsoft ATP has you covered.
  - organsnyder 6y ago
    Some of those have been true for quite some time. I ran my own mail server back in the 2000s to early 2010s, and incoming mail was especially hellish (effective spam filtering was basically impossible). These are really just variations on the theme of economies of scale, aren't they?
    - johnthescott 6y ago
      commercial dns black lists can be very effective at filtering spam and not expensive for personal use.
- hibikir 6y ago
  Having worked at some of those companies, along with small startups, the biggest difference isn't really plain old sophistication, but expenses so high as to make the engineering effort worthwhile. If your production system is 10 instances on some random cloud, a 10% efficiency savings saves me 1 instance, so maybe $2k a year. Taking into account opportunity costs vs doing things that raise revenue, said startup would consider the effort a waste of time unless it took a few hours. With the same architecture, but instead 20K instances, then suddenly that 10% is saving 2000 instances, and 4 million a year. Unless there's a major engineering shortage, chances are that spending a over a month on 4 million in yearly savings will be completely justified, and would even be a highlight in someone SRE's review. Chances are that the optimization wasn't even any harder in the big tech company: It's just that small savings on big piles of money are suddenly worth it. It's not possible to find millions of dollars in loose change between the proverbial couch cushions if you didn't have the chance to spend hundreds of millions in the first place. Heck, in a growth company, even at that size, 4 million might not be enough savings, as there might be even bigger things. Same with dev tooling: Saving a company 5% CI times is not great with 10 developers, but if you are making five thousand developers more productive, suddenly you can hire an entire team of very specialized developers, and it's not a luxury. This is also why often copying what large companies is foolish when you are small: The tradeoffs are going to be completely different. What would be an unacceptable flaw in a large company is just fine in your startup. The real trick IMO is the 200-800 range: Large enough that the simple solutions for small companies have probably broken down and are causing pain, but nowhere near the staffing to hire yourself a team to, say, add types to Ruby, get a team to build around the weaknesses of your database, or whichever other problem you have that any member of FAANG would just throw 10 million dollars in staffing costs without batting an eye. I've seen way too many companies that stopped being able to grow their company at those intermediate sizes, and get stuck in technical hell.
  - kabes 6y ago
    The problem is that managers at small companies with some, but insufficient, technical knowledge also read these articles and think they need to do the same. I come across overengineereed stuff at small companies all the time for things they don't need but only do because it's what Google/Facebook is doing.
- jacques_chester 6y ago
  If it makes you feel better, normal companies have "cost accountants", whose entire job is to worry about this kind of problem. Cost accounting a fairly well-developed field with a long history. There are several major approaches and some less well known ones. I think the most applicable in our field is "resource cost accounting", which looks superficially like what Netflix described. You keep "resource accounts" in the units of the actual stuff consumed by your business processes, then map backwards from the final product to costs. Of course I am not an accountant. Just took a course and found it enlightening.
  - jonplackett 6y ago
    I think the difference is that Netflix sees these problems and has the person-power to solve them and make themselves leaner. I guess it comes down to how far ahead you're looking really and tech companies are incentivised to look further ahead than your average big company. They're all planning for world domination (whether that actually happens or not)
    - jacques_chester 6y ago
      I don't disagree, I'm just trying to convey that in some sense the forward-looking effort would be accelerated by looking sideways as well. Cost accounting is practiced widely in many industries and appears in medium-sized businesses (tens of employees) as well as large businesses (tens of thousands of employees).
discodave 6y ago
Why on earth would Netflix want their blog behind a paywall? Absurd.
- thunderbong 6y ago
  I didn't get any paywall
  - aedocw 6y ago
    I see "You have 2 free stories left this month." at the top. It's a vanity URL for Medium. If they've got their own domain why would they want to drive any traffic to medium? That doesn't make any sense to me.
    - devwastaken 6y ago
      Probably red tape and sysadmins. Easier to just use a different service than it is to convince the sysadmins to host your blog software and maintain that.
      - dvtrn 6y ago
        Makes me wonder what made them choose Medium for their engineering blog versus your friendly neighborhood static website host.
        deleted 6y ago
        [deleted]
- timy2shoes 6y ago
  You can open it in incognito mode to avoid the paywall.
  - ramraj07 6y ago
    Problem solved, absurdity not solved.
gavinray 6y ago
Why does Netflix have half a dozen different database/database-esque systems? Snowflake, Hive, Cassandra, RDS, Druid, Presto Is there a good reason for this that people without experience at FAANG orgs can't grasp? To the naive, it seems like mixing Postgres, MySQL, Mongo, and Oracle. Also why is it on Medium under a Paywall, they are worth, quite literally, nearly $200 billion dollars =/
- dmlittle 6y ago
  Not sure about all of them but different databases have different use cases. It's not necessarily Postgres vs. MySQL where they are competing databases for the same use case. Snowflake, for example, is a data warehouse that (I believe) stores data in a columnar format. This makes it much better at performing aggregate queries (how many shows were watched on netflix today?) but a lot slower for specific queries (where in this episode is gavinray currently at?).
  - gavinray 6y ago
    Hive is a Data Warehouse that runs on top of Hadoop. Snowflake is a Data Warehouse/Lake, but it's also it's own custom SQL DB. Cassandra is a NoSQL DB RDS is running (some standard relational DB) Apache Druid is a columnar analytics DB centered around realtime uses. It has it's own query language and delegates to Apache Calcite for specific DB/datasource drivers. Can integrate with Kafka/Hadoop. Presto is (to my understanding) like a meta-DB that can query multiple databases. Similar to an integrated Apache Calcite or Google's ZetaSQL. There are a LOT of overlapping concerns here, which is why the confusion. Essentially they have 2-3 different products in several categories targeting generally the same usecase.
    - texasbigdata 6y ago
      Is there a good resource for this comparison plus the many streaming/ETL permutations. It's a bit confusing even just what the various Apache products do (beam, kylin, etc, etc).
    - nemothekid 6y ago
kohtatsu 6y ago
Hard sign-up walled :< https://i.imgur.com/mSk4uiI.png https://i.imgur.com/mSk4uiI.png
- kofejnik 6y ago
  nope; opened just fine in an anon chrome window with ublock origin
TheGuyWhoCodes 6y ago
Wow so a very large portion of their spend is on Cassandra. I bet it's heavily modified for their needs but I wonder if Scylladb would be able to replace some of those clusters for a considerable saving in maintenance time and money.
RcouF1uZ4gsC 6y ago
I wonder if Netflix is spending a lot of money and effort on things that consumers don't really care about. I don't really care for personalized recommendations. As long as the movies and TV Shows have some type of keyword or grouping, I am happy. Given the relatively fixed size of the catalog, and that most movies and TV shows are already classified (for example as Action or Romance or Comedy etc) this doesn't seem like something where a lot of data would do a lot to help. For me, most streaming movie companies are pretty equivalent in their experience. The only data I really need them to keep track of about me is what I have watched so I know if I have already watched something before, and where I am in the movie/tv series, so I can pick back up when I log in again. Everyone pretty much does it. As for streaming quality, basically everyone knows how to transcode movies and put them on a CDN. The size and quality of the catalog is the biggest differentiator for me.
- syspec 6y ago
  "I personally don't use feature X"
mrmonkeyman 6y ago
Focus on getting interesting content, Netflix. Literally the only thing worth talking about and also literally your core business. Recommendation 'systems' and metaserver bullshit: snore. Needing 8 DBMSes and 1K engineers to handle what amounts to extremely simplistic data: a sure sign they make too much money.
SergeAx 6y ago
Am I getting it right and Netflix' own hardware is negligible or non-existent and all their infrastructure is deployed to AWS?
- yencabulator 6y ago
  Netflix runs its own CDN system. That's a significant amount of hardware to own & manage, but their operating costs are probably less than what a conventional data center would have, because it's in the ISP's interest to host a local cache (to minimize bandwidth use). https://datacenterfrontier.com/mapping-netflix-content-delivery-network/ https://datacenterfrontier.com/mapping-netflix-content-deliv... [2016] https://openconnect.netflix.com/ https://openconnect.netflix.com/

132 points by el_duderino 6y ago

When I read something like this I realise just how immense of a lead tech companies are building vs 'normal' companies. There's just layer of expertise in how to streamline and organise themselves that anyone without a 1000 strong engineering team and huge engineering budget just are never going to think about.

ramraj07 6y ago
Ill argue that you don't need it for a small company - just don't overcomplicate your life by using technology that's designed for big companies. Stick to simple systems like elastic beanstalk or app engine + RDS and something like snowflake and you get powerful, manageable, auditable projects. I'm also finding great benefit in launching each project in its own AWS account. This means you know exactly what project costs how much!
- jonplackett 6y ago
  I was thinking more in terms of large but traditional companies. The kind tech companies put out of business regularly!
QUFB 6y ago
This is where things are heading, especially for business or corporate 'consumers': Need web hosting? Certainly you can't deal with a 2Tbps DDoS attack, so let's route that traffic through Cloudflare. Sending email? Can't send it out with those /23s you have, they're on some blacklist from a shady DNSBL provider. Need to pay for GSuite or Office365. Can't protect your CEO against a targeted attack? GSuite Advanced Protection or Microsoft ATP has you covered.
- organsnyder 6y ago
  Some of those have been true for quite some time. I ran my own mail server back in the 2000s to early 2010s, and incoming mail was especially hellish (effective spam filtering was basically impossible). These are really just variations on the theme of economies of scale, aren't they?
  - johnthescott 6y ago
    commercial dns black lists can be very effective at filtering spam and not expensive for personal use.
hibikir 6y ago
Having worked at some of those companies, along with small startups, the biggest difference isn't really plain old sophistication, but expenses so high as to make the engineering effort worthwhile. If your production system is 10 instances on some random cloud, a 10% efficiency savings saves me 1 instance, so maybe $2k a year. Taking into account opportunity costs vs doing things that raise revenue, said startup would consider the effort a waste of time unless it took a few hours. With the same architecture, but instead 20K instances, then suddenly that 10% is saving 2000 instances, and 4 million a year. Unless there's a major engineering shortage, chances are that spending a over a month on 4 million in yearly savings will be completely justified, and would even be a highlight in someone SRE's review. Chances are that the optimization wasn't even any harder in the big tech company: It's just that small savings on big piles of money are suddenly worth it. It's not possible to find millions of dollars in loose change between the proverbial couch cushions if you didn't have the chance to spend hundreds of millions in the first place. Heck, in a growth company, even at that size, 4 million might not be enough savings, as there might be even bigger things. Same with dev tooling: Saving a company 5% CI times is not great with 10 developers, but if you are making five thousand developers more productive, suddenly you can hire an entire team of very specialized developers, and it's not a luxury. This is also why often copying what large companies is foolish when you are small: The tradeoffs are going to be completely different. What would be an unacceptable flaw in a large company is just fine in your startup. The real trick IMO is the 200-800 range: Large enough that the simple solutions for small companies have probably broken down and are causing pain, but nowhere near the staffing to hire yourself a team to, say, add types to Ruby, get a team to build around the weaknesses of your database, or whichever other problem you have that any member of FAANG would just throw 10 million dollars in staffing costs without batting an eye. I've seen way too many companies that stopped being able to grow their company at those intermediate sizes, and get stuck in technical hell.
- kabes 6y ago
  The problem is that managers at small companies with some, but insufficient, technical knowledge also read these articles and think they need to do the same. I come across overengineereed stuff at small companies all the time for things they don't need but only do because it's what Google/Facebook is doing.
jacques_chester 6y ago
If it makes you feel better, normal companies have "cost accountants", whose entire job is to worry about this kind of problem. Cost accounting a fairly well-developed field with a long history. There are several major approaches and some less well known ones. I think the most applicable in our field is "resource cost accounting", which looks superficially like what Netflix described. You keep "resource accounts" in the units of the actual stuff consumed by your business processes, then map backwards from the final product to costs. Of course I am not an accountant. Just took a course and found it enlightening.
- jonplackett 6y ago
  I think the difference is that Netflix sees these problems and has the person-power to solve them and make themselves leaner. I guess it comes down to how far ahead you're looking really and tech companies are incentivised to look further ahead than your average big company. They're all planning for world domination (whether that actually happens or not)
  - jacques_chester 6y ago
    I don't disagree, I'm just trying to convey that in some sense the forward-looking effort would be accelerated by looking sideways as well. Cost accounting is practiced widely in many industries and appears in medium-sized businesses (tens of employees) as well as large businesses (tens of thousands of employees).

discodave 6y ago

Why on earth would Netflix want their blog behind a paywall? Absurd.

thunderbong 6y ago
I didn't get any paywall
- aedocw 6y ago
  I see "You have 2 free stories left this month." at the top. It's a vanity URL for Medium. If they've got their own domain why would they want to drive any traffic to medium? That doesn't make any sense to me.
  - devwastaken 6y ago
    Probably red tape and sysadmins. Easier to just use a different service than it is to convince the sysadmins to host your blog software and maintain that.
    - dvtrn 6y ago
      Makes me wonder what made them choose Medium for their engineering blog versus your friendly neighborhood static website host.
      - deleted 6y ago
        [deleted]
timy2shoes 6y ago
You can open it in incognito mode to avoid the paywall.
- ramraj07 6y ago
  Problem solved, absurdity not solved.

gavinray 6y ago

Why does Netflix have half a dozen different database/database-esque systems? Snowflake, Hive, Cassandra, RDS, Druid, Presto Is there a good reason for this that people without experience at FAANG orgs can't grasp? To the naive, it seems like mixing Postgres, MySQL, Mongo, and Oracle. Also why is it on Medium under a Paywall, they are worth, quite literally, nearly $200 billion dollars =/

dmlittle 6y ago
Not sure about all of them but different databases have different use cases. It's not necessarily Postgres vs. MySQL where they are competing databases for the same use case. Snowflake, for example, is a data warehouse that (I believe) stores data in a columnar format. This makes it much better at performing aggregate queries (how many shows were watched on netflix today?) but a lot slower for specific queries (where in this episode is gavinray currently at?).
- gavinray 6y ago
  Hive is a Data Warehouse that runs on top of Hadoop. Snowflake is a Data Warehouse/Lake, but it's also it's own custom SQL DB. Cassandra is a NoSQL DB RDS is running (some standard relational DB) Apache Druid is a columnar analytics DB centered around realtime uses. It has it's own query language and delegates to Apache Calcite for specific DB/datasource drivers. Can integrate with Kafka/Hadoop. Presto is (to my understanding) like a meta-DB that can query multiple databases. Similar to an integrated Apache Calcite or Google's ZetaSQL. There are a LOT of overlapping concerns here, which is why the confusion. Essentially they have 2-3 different products in several categories targeting generally the same usecase.
  - texasbigdata 6y ago
    Is there a good resource for this comparison plus the many streaming/ETL permutations. It's a bit confusing even just what the various Apache products do (beam, kylin, etc, etc).
  - nemothekid 6y ago

kohtatsu 6y ago

Hard sign-up walled :< https://i.imgur.com/mSk4uiI.png https://i.imgur.com/mSk4uiI.png

kofejnik 6y ago
nope; opened just fine in an anon chrome window with ublock origin

TheGuyWhoCodes 6y ago

Wow so a very large portion of their spend is on Cassandra. I bet it's heavily modified for their needs but I wonder if Scylladb would be able to replace some of those clusters for a considerable saving in maintenance time and money.

RcouF1uZ4gsC 6y ago

I wonder if Netflix is spending a lot of money and effort on things that consumers don't really care about. I don't really care for personalized recommendations. As long as the movies and TV Shows have some type of keyword or grouping, I am happy. Given the relatively fixed size of the catalog, and that most movies and TV shows are already classified (for example as Action or Romance or Comedy etc) this doesn't seem like something where a lot of data would do a lot to help. For me, most streaming movie companies are pretty equivalent in their experience. The only data I really need them to keep track of about me is what I have watched so I know if I have already watched something before, and where I am in the movie/tv series, so I can pick back up when I log in again. Everyone pretty much does it. As for streaming quality, basically everyone knows how to transcode movies and put them on a CDN. The size and quality of the catalog is the biggest differentiator for me.

syspec 6y ago
"I personally don't use feature X"

mrmonkeyman 6y ago

Focus on getting interesting content, Netflix. Literally the only thing worth talking about and also literally your core business. Recommendation 'systems' and metaserver bullshit: snore. Needing 8 DBMSes and 1K engineers to handle what amounts to extremely simplistic data: a sure sign they make too much money.

SergeAx 6y ago

Am I getting it right and Netflix' own hardware is negligible or non-existent and all their infrastructure is deployed to AWS?

yencabulator 6y ago
Netflix runs its own CDN system. That's a significant amount of hardware to own & manage, but their operating costs are probably less than what a conventional data center would have, because it's in the ISP's interest to host a local cache (to minimize bandwidth use). https://datacenterfrontier.com/mapping-netflix-content-delivery-network/ https://datacenterfrontier.com/mapping-netflix-content-deliv... [2016] https://openconnect.netflix.com/ https://openconnect.netflix.com/