14 ms·

Netflix Billing Migration to AWS – Part III

124 points by samber 10y ago

secabeen 10y ago
Netflix is well-known for running much of their infrastructure in AWS. What infrastructure do they not run in AWS? I'm aware of their CDN, which is the core of their business, and which they manage in-house. Are there other parts of the Netflix service that they self-host?
- jedberg 10y ago
  Not anymore, no. The billing was the last big thing to move. Everything except the CDN is on AWS (unless they moved something back to the datacenter since I left 1.5 years ago, but that's unlikely and I probably would have heard about it).
- keepper 10y ago
  It's pretty amazing Open Connect[1] doesn't get as much "fan fare" as Netflix use of AWS. This is arguably the core of what they do ( push tons of bits to the end user ), and something the cloud could not do for them. [1] https://openconnect.netflix.com/en/ https://openconnect.netflix.com/en/
  - chupy 10y ago
    I am curious if the other streaming video providers are doing something on the same principle to the Netflix Open Connect. Just by assuming that 5-10% of their actual Netflix video traffic gets served from AWS and the rest gets served from the Open Connect servers, it's probable that any other competitor (except Amazon Video) would see huge savings with a similar implementation.
merb 10y ago
last time I asked why not PostgreSQL, now I have my answer. I love netflix's posts. Way more than most other tech blogs and their responses on the how and why is mostly more accurate than others (looking at uber) especially their expertise in evaluating stuff is extremly good. What I especially wondered was why DRBD (i didn't had good experiences with it, neither with psql nor with mysql) guess with a lot of engineers + the hope to migrate to aurora at some day makes this a reasonable choice (i guess). Still funny that their whole billing infrastructure runs on like two nodes for a while (the masters) while I guess a lot of things there has a need for writes especially when the monthly billing takes place.
- brianwawok 10y ago
  You could send a monthly bill to the entire world in 2 servers I suspect.
  - Consultant32452 10y ago
    I wonder if you could bill the entire world from a single Raspberry Pi. I suspect it could be done.
- rodgerd 10y ago
  > now I have my answer. Yes, although it was interesting that they put themselves through ETL pain now for a future state they aren't moving to.
- philliphaydon 10y ago
  I had the same question running through my head. But I really like their simple justification. Completely honest. Not like an article recently posted.
jasonjei 10y ago
Does Netflix have an integration with the ERP? Is the ERP self-hosted, on cloud, or SaaS (like NetSuite)? Very curious to read if G/L is integrated or if transactions are posted manually on an end-of-month basis. Edit: it seems like the earlier blog post mentions there is a data feed to the GL, but leaves this rather vague. Very interested to read their ERP and data feed setup. What ERP does Netflix use too?
- lmickh 10y ago
  Remember reading years ago somewhere that it was Oracle ERP, but can't find a reference. That might have changed as they got closer to completing their cloud migration.
- burnstek 10y ago
  Workday.
Scarbutt 10y ago
With all the praise and hype postgresql gets, you wonder why all this awesome services are mysql first, RDS? mysql first, aurora? mysql compatible, google cloud sql? mysql, netflix? mysql, github? mysql. I think mysql users are just there quietly making money while postgres users are bashing mysql ;) I also think services like aurora are strong selling point for starting new tech businesses with mysql.
- lmickh 10y ago
  Better technical implementation != greater adoption. If it did, PHP wouldn't be where it is today (everywhere). Timing has more to do with adoption then the technical implementation. Being the first thing that is "good enough" for wide spread use is key. The last 20 years are riddled with examples such as Docker.
  - rimantas 10y ago
    Except for replication, which matters to big companies, PG cannot really claim better implementation. And when few use your product at scale it does not get tested in really stressful situation, and when it does we have Uber situation.
- hrayr 10y ago
  "...we did not have much in house expertise using PostgreSQL" This seems to be the underlying factor.
  - Scarbutt 10y ago
    That's actually my point.
    - philliphaydon 10y ago
      I don't see how that is your point.
- ben_jones 10y ago
hrayr 10y ago
It continues to surprise me how (why?) Netflix builds their infrastructure on their #1 competitors platform. They're basically funding Amazon Prime. They're undoubtedly smart, so they must have a lot of incentive in doing so.
- ebrewste 10y ago
  I would assume AWS allows Netflix to be lower cost, more agile, etc., allowing Netflix to deliver a better product for their customers and allowing them to focus on their core value (not servers). In addition, I doubt that Amazon Prime Video services go unfunded if Netflix wasn't an AWS customer. I don't think it's such a simple, two element zero sum concern.
  - gshulegaard 10y ago
    I think you are correct. And just because Netflix uses servers (hardware) hosted through AWS doesn't mean that they don't have plenty of proprietary in house tech (software) that powers their service. An example of such would be Open Connect (https://openconnect.itp.netflix.com/ https://openconnect.itp.netflix.com/) their CDN built on top of FreeBSD and Nginx (https://www.nginx.com/blog/why-netflix-chose-nginx-as-the-heart-of-its-cdn/ https://www.nginx.com/blog/why-netflix-chose-nginx-as-the-he...). But beyond software, content licensing and content creation are also major distinguishing factors in the streaming video space.
  - rebootthesystem 10y ago
    The better product part is debatable. These days we typically spend half an hour or more browsing to find something good to watch. The quality of titles available is terrible.
- OrwellianChild 10y ago
  AWS appears to be the best product for the job in terms of cost-effective scalability for their needs. That it happens to be a profit center for a competitor is a side concern. Furthermore, it's actually a benefit to the marketplace! It forces Netflix to differentiate it's product to retain customers/marketshare.
samlachance 10y ago
I would love to see that monthly bill...
- MrMullen 10y ago
  I wonder how all that works out for Netflix. They must get a massive discount but do they buy reserved instances? or are all of their instances reserved instances?
  - tomschlick 10y ago
    According to @jedberg... they don't get any discounts other than hitting those top level tiers which are discounted. They pay the same rates as us peasants for the lower levels. And yes, I'd imagine they heavily use reserved instances as their traffic patterns would be pretty predictable by now.
    - vidarh 10y ago
      I know people using a tiny little fraction of that which have negotiated discounts well beyond the published prices at usage well below the top tiers. If they do pay the public prices, they are being totally fleeced to the point where you'd have to ask what in the world they are doing.
      - tomschlick 10y ago
        Thats what I would have thought too... just going off of what he has said in previous comments directly to me.
        vidarh 10y ago
        I guess he could be technically right in that they might overall be so far into the "Contact Us" tier that they've just been given a steep enough discount for that part of their usage to nullify the premium on the lower tiers. Having Netflix people say they're paying the regular rates certainly is good for Amazon PR.
nstj 10y ago
> Some source tables had columns where Number meant an integer, in other cases it was used for decimal values, while some had really long values up to 38 digits. I'd be interested if anyone had any guesses as to what data type may require a 38 digit number.
- twoodfin 10y ago
  log2(10^38) ≈ 126. I bet these were 128 bit integers encoded in decimal.
  - nstj 10y ago
    Nice one, that sounds on the money :)
  - joneholland 10y ago
    UUIDs perhaps?
    - nstj 10y ago
      As Number values?
  - yeukhon 10y ago
    Can you elaborate on this? I have not seen this technique.
    - nstj 10y ago
      Not that I condone using an integer as a UUID, but here's some discussion around using a 128 bit integer to store one[0] [0]: http://stackoverflow.com/questions/18439520/is-there-a-128-bit-integer-in-c http://stackoverflow.com/questions/18439520/is-there-a-128-b...
    - deleted 10y ago
      [deleted]
homero 10y ago
I'm using chargebee and I love it
user5994461 10y ago
I'd love to see an estimation of NetFlix for running the same infrastructure on GCE. All the instances are cheaper and faster. The networking is substantially faster, especially on smaller instances (I can imagine they have a lot of network IO given their business). No more reserved instances to manage. The discount is automatic on GCE when instances ran 24/24 for the month. (Reserved instances are a management nightmare when you have many servers, you can actually loose money). Replace all spot instances with the equivalent on GCE, which has fix pricing (about 70% discount) and is more dependable. (I imagine the core video business is extremely spot friendly).

124 points by samber 10y ago

Netflix is well-known for running much of their infrastructure in AWS. What infrastructure do they not run in AWS? I'm aware of their CDN, which is the core of their business, and which they manage in-house. Are there other parts of the Netflix service that they self-host?

jedberg 10y ago
Not anymore, no. The billing was the last big thing to move. Everything except the CDN is on AWS (unless they moved something back to the datacenter since I left 1.5 years ago, but that's unlikely and I probably would have heard about it).
keepper 10y ago
It's pretty amazing Open Connect[1] doesn't get as much "fan fare" as Netflix use of AWS. This is arguably the core of what they do ( push tons of bits to the end user ), and something the cloud could not do for them. [1] https://openconnect.netflix.com/en/ https://openconnect.netflix.com/en/
- chupy 10y ago
  I am curious if the other streaming video providers are doing something on the same principle to the Netflix Open Connect. Just by assuming that 5-10% of their actual Netflix video traffic gets served from AWS and the rest gets served from the Open Connect servers, it's probable that any other competitor (except Amazon Video) would see huge savings with a similar implementation.

merb 10y ago

last time I asked why not PostgreSQL, now I have my answer. I love netflix's posts. Way more than most other tech blogs and their responses on the how and why is mostly more accurate than others (looking at uber) especially their expertise in evaluating stuff is extremly good. What I especially wondered was why DRBD (i didn't had good experiences with it, neither with psql nor with mysql) guess with a lot of engineers + the hope to migrate to aurora at some day makes this a reasonable choice (i guess). Still funny that their whole billing infrastructure runs on like two nodes for a while (the masters) while I guess a lot of things there has a need for writes especially when the monthly billing takes place.

brianwawok 10y ago
You could send a monthly bill to the entire world in 2 servers I suspect.
- Consultant32452 10y ago
  I wonder if you could bill the entire world from a single Raspberry Pi. I suspect it could be done.
rodgerd 10y ago
> now I have my answer. Yes, although it was interesting that they put themselves through ETL pain now for a future state they aren't moving to.
philliphaydon 10y ago
I had the same question running through my head. But I really like their simple justification. Completely honest. Not like an article recently posted.

jasonjei 10y ago

Does Netflix have an integration with the ERP? Is the ERP self-hosted, on cloud, or SaaS (like NetSuite)? Very curious to read if G/L is integrated or if transactions are posted manually on an end-of-month basis. Edit: it seems like the earlier blog post mentions there is a data feed to the GL, but leaves this rather vague. Very interested to read their ERP and data feed setup. What ERP does Netflix use too?

lmickh 10y ago
Remember reading years ago somewhere that it was Oracle ERP, but can't find a reference. That might have changed as they got closer to completing their cloud migration.
burnstek 10y ago
Workday.

Scarbutt 10y ago

With all the praise and hype postgresql gets, you wonder why all this awesome services are mysql first, RDS? mysql first, aurora? mysql compatible, google cloud sql? mysql, netflix? mysql, github? mysql. I think mysql users are just there quietly making money while postgres users are bashing mysql ;) I also think services like aurora are strong selling point for starting new tech businesses with mysql.

lmickh 10y ago
Better technical implementation != greater adoption. If it did, PHP wouldn't be where it is today (everywhere). Timing has more to do with adoption then the technical implementation. Being the first thing that is "good enough" for wide spread use is key. The last 20 years are riddled with examples such as Docker.
- rimantas 10y ago
  Except for replication, which matters to big companies, PG cannot really claim better implementation. And when few use your product at scale it does not get tested in really stressful situation, and when it does we have Uber situation.
hrayr 10y ago
"...we did not have much in house expertise using PostgreSQL" This seems to be the underlying factor.
- Scarbutt 10y ago
  That's actually my point.
  - philliphaydon 10y ago
    I don't see how that is your point.
ben_jones 10y ago

hrayr 10y ago

It continues to surprise me how (why?) Netflix builds their infrastructure on their #1 competitors platform. They're basically funding Amazon Prime. They're undoubtedly smart, so they must have a lot of incentive in doing so.

ebrewste 10y ago
I would assume AWS allows Netflix to be lower cost, more agile, etc., allowing Netflix to deliver a better product for their customers and allowing them to focus on their core value (not servers). In addition, I doubt that Amazon Prime Video services go unfunded if Netflix wasn't an AWS customer. I don't think it's such a simple, two element zero sum concern.
- gshulegaard 10y ago
  I think you are correct. And just because Netflix uses servers (hardware) hosted through AWS doesn't mean that they don't have plenty of proprietary in house tech (software) that powers their service. An example of such would be Open Connect (https://openconnect.itp.netflix.com/ https://openconnect.itp.netflix.com/) their CDN built on top of FreeBSD and Nginx (https://www.nginx.com/blog/why-netflix-chose-nginx-as-the-heart-of-its-cdn/ https://www.nginx.com/blog/why-netflix-chose-nginx-as-the-he...). But beyond software, content licensing and content creation are also major distinguishing factors in the streaming video space.
- rebootthesystem 10y ago
  The better product part is debatable. These days we typically spend half an hour or more browsing to find something good to watch. The quality of titles available is terrible.
OrwellianChild 10y ago
AWS appears to be the best product for the job in terms of cost-effective scalability for their needs. That it happens to be a profit center for a competitor is a side concern. Furthermore, it's actually a benefit to the marketplace! It forces Netflix to differentiate it's product to retain customers/marketshare.

samlachance 10y ago

I would love to see that monthly bill...

MrMullen 10y ago
I wonder how all that works out for Netflix. They must get a massive discount but do they buy reserved instances? or are all of their instances reserved instances?
- tomschlick 10y ago
  According to @jedberg... they don't get any discounts other than hitting those top level tiers which are discounted. They pay the same rates as us peasants for the lower levels. And yes, I'd imagine they heavily use reserved instances as their traffic patterns would be pretty predictable by now.
  - vidarh 10y ago
    I know people using a tiny little fraction of that which have negotiated discounts well beyond the published prices at usage well below the top tiers. If they do pay the public prices, they are being totally fleeced to the point where you'd have to ask what in the world they are doing.
    - tomschlick 10y ago
      Thats what I would have thought too... just going off of what he has said in previous comments directly to me.
      - vidarh 10y ago
        I guess he could be technically right in that they might overall be so far into the "Contact Us" tier that they've just been given a steep enough discount for that part of their usage to nullify the premium on the lower tiers. Having Netflix people say they're paying the regular rates certainly is good for Amazon PR.

nstj 10y ago

> Some source tables had columns where Number meant an integer, in other cases it was used for decimal values, while some had really long values up to 38 digits. I'd be interested if anyone had any guesses as to what data type may require a 38 digit number.

twoodfin 10y ago
log2(10^38) ≈ 126. I bet these were 128 bit integers encoded in decimal.
- nstj 10y ago
  Nice one, that sounds on the money :)
- joneholland 10y ago
  UUIDs perhaps?
  - nstj 10y ago
    As Number values?
- yeukhon 10y ago
  Can you elaborate on this? I have not seen this technique.
  - nstj 10y ago
    Not that I condone using an integer as a UUID, but here's some discussion around using a 128 bit integer to store one[0] [0]: http://stackoverflow.com/questions/18439520/is-there-a-128-bit-integer-in-c http://stackoverflow.com/questions/18439520/is-there-a-128-b...
  - deleted 10y ago
    [deleted]

homero 10y ago

I'm using chargebee and I love it

user5994461 10y ago

I'd love to see an estimation of NetFlix for running the same infrastructure on GCE. All the instances are cheaper and faster. The networking is substantially faster, especially on smaller instances (I can imagine they have a lot of network IO given their business). No more reserved instances to manage. The discount is automatic on GCE when instances ran 24/24 for the month. (Reserved instances are a management nightmare when you have many servers, you can actually loose money). Replace all spot instances with the equivalent on GCE, which has fix pricing (about 70% discount) and is more dependable. (I imagine the core video business is extremely spot friendly).