5 ms·

Why Netflix Chose Amazon’s Cloud as Their Computing Platform

55 points by tomeast 16y ago

redwood 16y ago
Amazon has plenty of spare cycles most of the year (hence Mechanical Turk and all this excess datacenter capacity they can contract out). - They have this b/c they need to be able to accommodate their extrmeley peaky demand (i.e. holiday shopping season!!!) - I wonder if this means netflix streaming will be significantly for flaky in the run-up to Xmas :)
- wmf 16y ago
  EC2 doesn't run on spare capacity (although perhaps Amazon buys from EC2 in competition with other customers) and Netflix streaming doesn't run on EC2 anyway.
maukdaddy 16y ago
We could have chosen to build out new data centers, build our own redundancy and failover, data synchronization systems, etc. Or, we could opt to write a check to someone else to do that instead. If you are building almost any type of company today, it makes no sense to build your own datacenter. We're going to look back in 10-20 years and laugh that every company felt the need to build datacenters. While doing some research relating to sustainability and cloud computing, I found some statistics showing that federal agencies saved between 25-50% by moving to "cloud" solutions. http://www.brookings.edu/papers/2010/0407_cloud_computing_west.aspx http://www.brookings.edu/papers/2010/0407_cloud_computing_we... Cloud computing is an excellent way for companies to dramatically lower their carbon footprint as well. Instead of using coal-fired electricity running your own inefficient datacenters, why not host your stuff at Google or Amazon where the PUE is low and they're using hydro/cleaner power?
- chime 16y ago
  > We're going to look back in 10-20 years and laugh that every company felt the need to build datacenters. Until recently, I don't think it was "felt the need to" as much as "had no choice but to." AWS-style data-center is a new development in technology. The ability to remotely fire up servers and have them running in minutes is wonderful but it also means you have to redo your entire IT architecture. For companies just starting up, it's no big deal. For ones already established for 30 years, it is nothing short of a Herculean effort. Setting up a datacenter might be a better choice.
  - maukdaddy 16y ago
    Very good point, and a poor choice of words on my part. OTOH, there are companies now that can handle mainframe SaaS ( outsourcing your mainframe apps). Huge gains for companies that want to take the leap and dismantle their mainframes.
- RyanGWU82 16y ago
  If you are building almost any type of company today, it makes no sense to build your own datacenter. Except for when it _does_ make sense. I'm starting a project at my company to add a single server in-house, and move some data storage away from Amazon S3. This will save us thousands of dollars a month -- the payback period for the project is 3 months. (!) That includes the engineering cost to set up the new server. This particular project is compelling because we don't need S3's other advantages, we just need raw storage. Moral: Don't be dogmatic about decisions in your business. Just weigh the pros and cons (including cost) carefully.
  - maukdaddy 16y ago
    Hence the reason I said "almost". There will always be exceptions. When looking at a cost/benefit don't forget the following costs: Power Cooling Space (lease costs) Administration Backup Redundancy Decommissioning (you don't just throw the server away do you?)
  - foobarbazetc 16y ago
    Price/performance is absolutely terrible on AWS. What most people don't realize and/or talk about with AWS is how much more expensive it is than in-house/co-lo/dedicated elsewhere.
- nradov 16y ago
  HIPAA privacy and security rules are still a significant obstacle for hosting US healthcare applications on third-party clouds. Those obstacles will eventually fade away as the covered entities become more comfortable with cloud security, and the cloud providers offer standardized business associate agreements. But it may take 10 years or more until we see widespread adoption.
  - cwinters 16y ago
    Have you tried to go through this process? Amazon seems to have done a good deal of work to ensure its servers and practices are HIPAA-compliant: http://aws.amazon.com/about-aws/whats-new/2009/04/06/whitepaper-hipaa/ http://aws.amazon.com/about-aws/whats-new/2009/04/06/whitepa... I think, as you allude, that the primary issue is not technical but behavioral. In the area of healthcare I work in (nursing homes), quite a few organizations have their data hosted elsewhere. Sometimes it's through the vendor, sometimes it's with a third-party service. I don't see that big a leap of faith to Amazon from this.
    - nradov 16y ago
      I haven't tried. Our prospective customers often want to physically inspect our data center as part of their due diligence process, and I can't see how Amazon could accommodate that.
- akg_67 16y ago
  After what Amazon and other US providers did to WikiLeaks, I will recommend everyone to have a backup plan to outsourcing data center to 'cloud'. And, backup doesn't mean another instance on Amazon cloud or another similar cloud provider.
- ars 16y ago
  I think you mean a higher PUE. Anyway, a better PUE will certainly help your carbon footprint, but saying "using hydro/cleaner power" is meaningless. You don't get to pick and choose "I'm using hydro, you're using coal" - the power grid is interconnected - everyone is using everything. Additionally all the possible hydro that can be used already is, so it's not like you can individually promote hydro usage - if you "use" hydro power, all the happens is that someone else doesn't get to use it, which means effectively you are burning coal, but remotely. There are some slight benefits to using the power near where it's generated to avoid transmission losses, but it's not large. Unless you can say "because of me there is a new, cleaner, power station that is built" you are not using cleaner power, you are just shifting less clean power to someone else.
byteclub 16y ago
I think what that post is trying to say is this: focus on your core competency. Amazon is good at cloud computing, Netflix is good at renting/streaming movies. However, there are some cases where "core competency" is a vague term. Take Facebook for example - could they outsource their infrastructure to somebody else? Given how much data they have to handle, probably not. When you get to the point where scaling requires very low-level (Hunch) or massive (Google or Facebook) optimizations/deployments, you might be entering the business of handling intricate infrastructure yourself.
- SRSimko 16y ago
  I work at an IaaS company and we hear the same thing daily most want to focus on what they do best and often it's not server hugging. Also companies that embrace the cloud have changed how they do things because they see the efficiencies and recognize that they don't want to be in the business of managing servers.
shimi 16y ago
Netflix have done the same thing with Roku, they like to focus on what they do best
somic 16y ago
This is indeed an interesting post but it doesn't answer the question why they chose Amazon. It focuses on why they chose not to expand within their own datacenter.
- jeremyjarvis 16y ago
  Agreed, I was interested to see which of Amazon's features were particularly of use to them. As opposed to, say, Rackspace Cloud.
  - somic 16y ago
    This talk by @adrianco may address the points you were looking for - http://www.blip.tv/file/4252897 http://www.blip.tv/file/4252897 To sum up, Netflix's strategy is to go with the best and biggest in each category, and AWS is obviously the biggest as of now.
    - brown9-2 16y ago
      If anyone is interested slides from that talk are here: http://www.slideshare.net/adrianco/netflix-on-cloud-combined-slides-for-dev-and-ops http://www.slideshare.net/adrianco/netflix-on-cloud-combined...
- pchristensen 16y ago
  Who would be a viable alternative vendor to Amazon at Netflix's scale?
  - sgk284 16y ago
    Rackspace?
epoxyhockey 16y ago
I'm still apprehensive about entrusting a single party to handle all of my infrastructure. Technical and business failures occur in even the best of the hosting companies. Granted, my computing needs are basic compared to the needs of Netflix.
- wmf 16y ago
  In-house IT is also a single party.
  - epoxyhockey 16y ago
    I think the motivation for in-house IT to succeed is greater than a 3rd party with their SLA. But, I am also coming from the viewpoint that geographically dispersed servers through multiple hosting companies reduces the 'single point of failure' scenario.
- ewams 16y ago
  Using one (or a mix) of hardware vendors entrusts your infrastructure to a single party as well. Unless you are making your own hardware?
frossie 16y ago
"We’re not very good at predicting customer growth or device engagement." But they are good enough to admit that, and plan accordingly. Hat off.
bdb 16y ago
What happens when Amazon launches their competing streaming video service? [1] Just a rumor at this point, but when a large set of content comes in-window for both Netflix and Amazon and each needs to spin up a thousand instances to transcode it? Who will get priority? [2] It seems extraordinarily stupid to entrust such an obvious potential competitor with the keys to your infrastructure, particularly in a single-vendor solution. This is fundamentally different from the Roku model -- neither Amazon nor Netflix own Roku, so they can remain neutral. (edit: formatting) [1] http://technologizer.com/2010/08/31/amazon-subscription-streaming-rumor/ http://technologizer.com/2010/08/31/amazon-subscription-stre... [2] And don't give me any bullshit about AWS not being capacity constrained unless you routinely spin up 1000 temporary XL instances.
- samratjp 16y ago
  "It seems extraordinarily stupid to entrust such an obvious potential competitor with the keys to your infrastructure, particularly in a single-vendor solution." If this (http://www.google.com/finance?client=ob&q=NASDAQ:NFLX http://www.google.com/finance?client=ob&q=NASDAQ:NFLX) is any indication, I am sure they are secretly building their farm somewhere as we speak. On a side note, I wonder why all these companies brag about AWS so much (yeah, yeah, AWS is awesome, I use it myself too) - are they getting endorsement incentives or something?
  - bdb 16y ago
    From what I have heard, they are not. They are literally going full speed ahead moving all of their infrastructure (aside of streaming) over to AWS.
  - brown9-2 16y ago
    Netflix has recently started a "tech blog" and is keeping it regularly updated with articles, although so far they mostly seem to be about "How Netflix uses tech" rather than talking about their code, OSS projects, low-level design decisions, etc. I think they're starting to discuss these things a bit more as a recruitment strategy.
tcskeptic 16y ago
I look in on this stuff from the enterprise side of things with interest. I want to get out of the business of hosting an internal datacenter entirely as soon as possible (that decades old, and apparently intractable AS400 accounting system notwithstanding), and would prefer to move things cloudward. Right now though, it seems that we are either limited to niche players or proprietary solutions(AWS, etc.), and the risk of being held hostage by an underperforming vendor remains high. We are experimenting with certain applications (CRM, for example) being hosted in the cloud. Until there is a standard cloud-hosting platform, with multiple large players offering services that would allow us to very simply move from one service to another without redisigning major parts of applications there won't be much movement to the cloud in our space. (part of this is a legacy of horrendous application design in the enterprise space, part of it is just the natural legacy of having much older legacy applications than web companies do, and part of it is a reflection of large organizations being terribly risk averse.)

55 points by tomeast 16y ago

Amazon has plenty of spare cycles most of the year (hence Mechanical Turk and all this excess datacenter capacity they can contract out). - They have this b/c they need to be able to accommodate their extrmeley peaky demand (i.e. holiday shopping season!!!) - I wonder if this means netflix streaming will be significantly for flaky in the run-up to Xmas :)

wmf 16y ago
EC2 doesn't run on spare capacity (although perhaps Amazon buys from EC2 in competition with other customers) and Netflix streaming doesn't run on EC2 anyway.

maukdaddy 16y ago

We could have chosen to build out new data centers, build our own redundancy and failover, data synchronization systems, etc. Or, we could opt to write a check to someone else to do that instead. If you are building almost any type of company today, it makes no sense to build your own datacenter. We're going to look back in 10-20 years and laugh that every company felt the need to build datacenters. While doing some research relating to sustainability and cloud computing, I found some statistics showing that federal agencies saved between 25-50% by moving to "cloud" solutions. http://www.brookings.edu/papers/2010/0407_cloud_computing_west.aspx http://www.brookings.edu/papers/2010/0407_cloud_computing_we... Cloud computing is an excellent way for companies to dramatically lower their carbon footprint as well. Instead of using coal-fired electricity running your own inefficient datacenters, why not host your stuff at Google or Amazon where the PUE is low and they're using hydro/cleaner power?

chime 16y ago
> We're going to look back in 10-20 years and laugh that every company felt the need to build datacenters. Until recently, I don't think it was "felt the need to" as much as "had no choice but to." AWS-style data-center is a new development in technology. The ability to remotely fire up servers and have them running in minutes is wonderful but it also means you have to redo your entire IT architecture. For companies just starting up, it's no big deal. For ones already established for 30 years, it is nothing short of a Herculean effort. Setting up a datacenter might be a better choice.
- maukdaddy 16y ago
  Very good point, and a poor choice of words on my part. OTOH, there are companies now that can handle mainframe SaaS ( outsourcing your mainframe apps). Huge gains for companies that want to take the leap and dismantle their mainframes.
RyanGWU82 16y ago
If you are building almost any type of company today, it makes no sense to build your own datacenter. Except for when it _does_ make sense. I'm starting a project at my company to add a single server in-house, and move some data storage away from Amazon S3. This will save us thousands of dollars a month -- the payback period for the project is 3 months. (!) That includes the engineering cost to set up the new server. This particular project is compelling because we don't need S3's other advantages, we just need raw storage. Moral: Don't be dogmatic about decisions in your business. Just weigh the pros and cons (including cost) carefully.
- maukdaddy 16y ago
  Hence the reason I said "almost". There will always be exceptions. When looking at a cost/benefit don't forget the following costs: Power Cooling Space (lease costs) Administration Backup Redundancy Decommissioning (you don't just throw the server away do you?)
- foobarbazetc 16y ago
  Price/performance is absolutely terrible on AWS. What most people don't realize and/or talk about with AWS is how much more expensive it is than in-house/co-lo/dedicated elsewhere.
nradov 16y ago
HIPAA privacy and security rules are still a significant obstacle for hosting US healthcare applications on third-party clouds. Those obstacles will eventually fade away as the covered entities become more comfortable with cloud security, and the cloud providers offer standardized business associate agreements. But it may take 10 years or more until we see widespread adoption.
- cwinters 16y ago
  Have you tried to go through this process? Amazon seems to have done a good deal of work to ensure its servers and practices are HIPAA-compliant: http://aws.amazon.com/about-aws/whats-new/2009/04/06/whitepaper-hipaa/ http://aws.amazon.com/about-aws/whats-new/2009/04/06/whitepa... I think, as you allude, that the primary issue is not technical but behavioral. In the area of healthcare I work in (nursing homes), quite a few organizations have their data hosted elsewhere. Sometimes it's through the vendor, sometimes it's with a third-party service. I don't see that big a leap of faith to Amazon from this.
  - nradov 16y ago
    I haven't tried. Our prospective customers often want to physically inspect our data center as part of their due diligence process, and I can't see how Amazon could accommodate that.
akg_67 16y ago
After what Amazon and other US providers did to WikiLeaks, I will recommend everyone to have a backup plan to outsourcing data center to 'cloud'. And, backup doesn't mean another instance on Amazon cloud or another similar cloud provider.
ars 16y ago
I think you mean a higher PUE. Anyway, a better PUE will certainly help your carbon footprint, but saying "using hydro/cleaner power" is meaningless. You don't get to pick and choose "I'm using hydro, you're using coal" - the power grid is interconnected - everyone is using everything. Additionally all the possible hydro that can be used already is, so it's not like you can individually promote hydro usage - if you "use" hydro power, all the happens is that someone else doesn't get to use it, which means effectively you are burning coal, but remotely. There are some slight benefits to using the power near where it's generated to avoid transmission losses, but it's not large. Unless you can say "because of me there is a new, cleaner, power station that is built" you are not using cleaner power, you are just shifting less clean power to someone else.

byteclub 16y ago

I think what that post is trying to say is this: focus on your core competency. Amazon is good at cloud computing, Netflix is good at renting/streaming movies. However, there are some cases where "core competency" is a vague term. Take Facebook for example - could they outsource their infrastructure to somebody else? Given how much data they have to handle, probably not. When you get to the point where scaling requires very low-level (Hunch) or massive (Google or Facebook) optimizations/deployments, you might be entering the business of handling intricate infrastructure yourself.

SRSimko 16y ago
I work at an IaaS company and we hear the same thing daily most want to focus on what they do best and often it's not server hugging. Also companies that embrace the cloud have changed how they do things because they see the efficiencies and recognize that they don't want to be in the business of managing servers.

shimi 16y ago

Netflix have done the same thing with Roku, they like to focus on what they do best

somic 16y ago

This is indeed an interesting post but it doesn't answer the question why they chose Amazon. It focuses on why they chose not to expand within their own datacenter.

jeremyjarvis 16y ago
Agreed, I was interested to see which of Amazon's features were particularly of use to them. As opposed to, say, Rackspace Cloud.
- somic 16y ago
  This talk by @adrianco may address the points you were looking for - http://www.blip.tv/file/4252897 http://www.blip.tv/file/4252897 To sum up, Netflix's strategy is to go with the best and biggest in each category, and AWS is obviously the biggest as of now.
  - brown9-2 16y ago
    If anyone is interested slides from that talk are here: http://www.slideshare.net/adrianco/netflix-on-cloud-combined-slides-for-dev-and-ops http://www.slideshare.net/adrianco/netflix-on-cloud-combined...
pchristensen 16y ago
Who would be a viable alternative vendor to Amazon at Netflix's scale?
- sgk284 16y ago
  Rackspace?

epoxyhockey 16y ago

I'm still apprehensive about entrusting a single party to handle all of my infrastructure. Technical and business failures occur in even the best of the hosting companies. Granted, my computing needs are basic compared to the needs of Netflix.

wmf 16y ago
In-house IT is also a single party.
- epoxyhockey 16y ago
  I think the motivation for in-house IT to succeed is greater than a 3rd party with their SLA. But, I am also coming from the viewpoint that geographically dispersed servers through multiple hosting companies reduces the 'single point of failure' scenario.
ewams 16y ago
Using one (or a mix) of hardware vendors entrusts your infrastructure to a single party as well. Unless you are making your own hardware?

frossie 16y ago

"We’re not very good at predicting customer growth or device engagement." But they are good enough to admit that, and plan accordingly. Hat off.

bdb 16y ago

What happens when Amazon launches their competing streaming video service? [1] Just a rumor at this point, but when a large set of content comes in-window for both Netflix and Amazon and each needs to spin up a thousand instances to transcode it? Who will get priority? [2] It seems extraordinarily stupid to entrust such an obvious potential competitor with the keys to your infrastructure, particularly in a single-vendor solution. This is fundamentally different from the Roku model -- neither Amazon nor Netflix own Roku, so they can remain neutral. (edit: formatting) [1] http://technologizer.com/2010/08/31/amazon-subscription-streaming-rumor/ http://technologizer.com/2010/08/31/amazon-subscription-stre... [2] And don't give me any bullshit about AWS not being capacity constrained unless you routinely spin up 1000 temporary XL instances.

samratjp 16y ago
"It seems extraordinarily stupid to entrust such an obvious potential competitor with the keys to your infrastructure, particularly in a single-vendor solution." If this (http://www.google.com/finance?client=ob&q=NASDAQ:NFLX http://www.google.com/finance?client=ob&q=NASDAQ:NFLX) is any indication, I am sure they are secretly building their farm somewhere as we speak. On a side note, I wonder why all these companies brag about AWS so much (yeah, yeah, AWS is awesome, I use it myself too) - are they getting endorsement incentives or something?
- bdb 16y ago
  From what I have heard, they are not. They are literally going full speed ahead moving all of their infrastructure (aside of streaming) over to AWS.
- brown9-2 16y ago
  Netflix has recently started a "tech blog" and is keeping it regularly updated with articles, although so far they mostly seem to be about "How Netflix uses tech" rather than talking about their code, OSS projects, low-level design decisions, etc. I think they're starting to discuss these things a bit more as a recruitment strategy.

tcskeptic 16y ago

I look in on this stuff from the enterprise side of things with interest. I want to get out of the business of hosting an internal datacenter entirely as soon as possible (that decades old, and apparently intractable AS400 accounting system notwithstanding), and would prefer to move things cloudward. Right now though, it seems that we are either limited to niche players or proprietary solutions(AWS, etc.), and the risk of being held hostage by an underperforming vendor remains high. We are experimenting with certain applications (CRM, for example) being hosted in the cloud. Until there is a standard cloud-hosting platform, with multiple large players offering services that would allow us to very simply move from one service to another without redisigning major parts of applications there won't be much movement to the cloud in our space. (part of this is a legacy of horrendous application design in the enterprise space, part of it is just the natural legacy of having much older legacy applications than web companies do, and part of it is a reflection of large organizations being terribly risk averse.)