11 ms·

Netflix, Long an AWS Customer, Tests Waters on Google Cloud

233 points by raleighm 9y ago

Aissen 9y ago
Looks like this is paywalled. Edit: Since this is being downvoted, here is the proof: https://screenshots.firefox.com/bgD6FxWvQS4e0sB2/www.theinformation.com https://screenshots.firefox.com/bgD6FxWvQS4e0sB2/www.theinfo... I have no issue with paywalled-content, it should just be noted that this is, at least for some readers.
- petercooper 9y ago
  I agree. If putting [video] on video links is a thing here, so should [paywall] or similar since it's even more annoying than videos. However, I'm a subscriber to The Information so here's a share link: https://go.theinformation.com/f2b133baedc57a8c https://go.theinformation.com/f2b133baedc57a8c - I think it requres an email, alas, but it's something if people are desperate to read the story.
blackaspen 9y ago
A lot of this article isn't that surprising -- especially the part where they're keeping "Business Critical" data on GCP. Of course they are! Every company needs a disaster recovery plan and in general shouldn't put their eggs in one basket. I'd be interested to see where the AI/ML frameworks differ though, and what additional value one can extract using GCP versus AWS and at what cost/presumptions.
forgot-my-pw 9y ago
I would think if a business use more than 500 servers, it would make sense to start considering having your own datacenter?
- blackaspen 9y ago
  It's different for everyone. If your 500 servers are for storage, S3/GCS will beat you in cost. For compute, that might be a different story, but then you still have to consider regionality, networking, etc.
  - supernovae 9y ago
    Let's be honest, even in the public cloud there are very few orgs that can survive regional failures and most people solve locality through CDN's for content delivery which means a private datacenter isn't really that different from public cloud for most orgs. Also, when when orgs move to the cloud they don't fire their networking team, they still have deskopts/laptops/servers/security devices/printers and large lans/wans to connect / peer already and these teams could staff up for private datacenter without really adding additional costs that aren't already incurred as part of doing business anyway. Our private datacenter is a few thousand servers leased space in a colo and we have one "datacenter dude" who handles hardware issues and our network team spends 90% of their time still on corporate infra vs actual supports with colo space. One can take some up front engineering/planning costs to scope out a private datacenter for about the same cost as it is to scope/plan VPC's and peering/connectivity/vpn services.
    - blackaspen 9y ago
      I completely agree. I jokingly tell everyone I know using AWS to "not use us-east1" and that they need to use another region not just AZs...and they always laugh and say "you were right" with us-east goes down. At least from my view, there's a difference in backbone versus corporate space networking, but you're right, most everyone won't ever realize any savings from that side of things.
      - jtwaleson 9y ago
        How often does us-east-1 go down? I know of about 4 big outages in the last 7 years but you're making it sound like it happens much more frequently. Is there a list of outages somewhere?
        supernovae 9y ago
        it's not just about it going down.. if you rely only on us-east they have had multiple times when you couldn't autoscale because there was no more capacity... if you rely on running lean and scale/up/scale down events us-east would have bitten your ass a lot the past year... Just for comparison sake even our private colo has had 0 issues the past decade that caused any interruptions to service
- wise0wl 9y ago
  You really have to factor in the labor costs of infrastructure and operations into that as well. Running your own private "cloud" isn't cheap. And no matter what you do, you aren't going to get even close to feature parity with the public cloud offerings (or even speed, if you are using virtual-machines). There are a lot of trade offs that you have to weigh in that don't always come down to bare dollars and cents.
  - ocdtrekkie 9y ago
    Everything you'd have to pay for, Amazon or Google has to pay for for you... plus profit margin. At Netflix scale, it's almost impossible for someone else's cloud to be cheaper than running your own, and hiring the necessary staff and acquiring the necessary space and hardware to do so. I honestly have to wonder if the level of Amazon's discount for Netflix is absurd to the tune of "we lose money doing this" just to retain them. AWS may need Netflix, to make serving it's smaller clients economically viable, more than Netflix needs AWS.
    - weego 9y ago
      If you're a margin business then certainly. If you're not then there's a lot to be said for keeping your core competency lean and offloading the heavy lifting at a negotiated price.
    - drstewart 9y ago
      This is assuming you can do everything at the same efficiency level as Amazon or Google. It's likely that you wouldn't be able to outperform on these metrics to a degree that it would make the profit margin difference a wash.
      - ocdtrekkie 9y ago
        The exact point I am making is that at a certain point, you are able to do things as efficiently as they are. Netflix certainly qualifies at that scale.
- supernovae 9y ago
  I wouldn't do my own datacenter as in build out a building, but I would certainly lease a cage and build out my own infrastructure at probably half that count. As far as I know, Netflix has POP's for content delivery in many carrier facilities so it's not a pure cloud player by any means. For the cost of 1 year for a single cloud server you can buy a 52 core dell with 256gb ram and a few SSD's and have price/performance ratios that heavily scale in your favor and with a scheduler like marathon, k8s, nomad you can easily achieve elasticity in a private colo to a large degree without also marrying yourself to cloud specifics and nothing would stop you from peering with major cloud providers for seasonal elastic needs.
  - blackaspen 9y ago
    That's a good distinction. CoLo providers are pretty reasonably priced and most of the larger ones are good at what they do. There's a HUGE scale difference between a CoLo and your own -- as soon as you start having to negotiate diesel supply contracts yourself things get crazy fast.
    - supernovae 9y ago
      I would never roll my own unless I was of facebook/google size. Colo space is abundant and affordable.
  - willshepherdson 9y ago
    I think this is a crucial distinction. Netflix has a relatively unique architecture for their content delivery system compared to other online video providers, which is worth considering in context of this article. With Netflix's Open Connect (https://openconnect.netflix.com/en/ https://openconnect.netflix.com/en/) system, most residential Internet Service Providers have appliances at the regional point of interconnection. These appliances are inside of or directly connected to the ISP's network in the region. Back in 2016 they had appliances deployed to over 1,000 locations. Content is pre-positioned during off-peak hours to the appliances. Previously Netflix has been open about how they use AWS for "generic, scalable computing" including "all of the logic of the application interface, the content discovery and selection experience, recommendation algorithms, transcoding, etc." At the time, they claimed that one of the advantages of doing this was the ease of use and growing commoditization of the “cloud” market, so it makes sense that they are evaluating alternative cloud providers for these commodity services.
- bluejekyll 9y ago
  This isn’t such an obvious decision. Running your own datacenter is difficult. Besides racking and stacking, you need to deal with power and cooling. Most likely you will outsource all of this to a collocations management services initially, which becomes your overhead just like AWS/Google. But they have economies of scale, in addition, they have expertise in managing all of this equipment with very high reliability. They also have tons of spare capacity (elastic) that are at your disposal, you’ll lose this in your own datacenter. On top of that you need to hire specialists, and 24/7 noc teams, for making sure your site is always up. And it’s not just one, you need a minimum of two, geographically separated so that they are not effected by the same fire, earthquake, power outage from ice/wind/electrical storms. At the end of the day, You need to beat AWS’ profit margin, and you still would have a huge amount of difficulty being able to match the uptime that you get when you’re so easily able to distribute services across regions and availability zones that are resilient to all the real world problems listed above. And all of those do happen. Even for the largest of companies, I’m not sure it makes sense to compete with these services except in the most niche of markets (like dropbox and storage).
  - supernovae 9y ago
    The cloud doesn't really have economies of scale unless your workloads scale opposite of pricing - which they very rarely do. For small shops, sure, but for the example of 500 instances running - you could lease space and grow with physical servers at a better economic scale. If you only need 500 servers for 1 busy season and can drop back down to 100 then stay in the cloud. If you need 500 always running instances and your growth is 20% year over year, i can almost guarantee a good colo will be cheaper and faster if you use the same automation and configuration as code philosophies in private dc as you would in public cloud. Puppet/foreman/cobbler/ansible/chef all work great in private DC
    - bluejekyll 9y ago
      In general, I think it’s a wash. The margins you save on end up costing you in other ways, such as development velocity and having access to easily deployed high-scale solutions like dynamo and managed instances like RDS. it takes time and organizational know-how to get to a point where similar things can be managed in the same way as AWS. Again, there are niche segments where this might not be true, 500 constant machines churning through massive datasets all day-long, etc, might fall into that category, but it’s not slam dunk.
- jwilliams 9y ago
  It's a good question, but depends on your workload. Netflix is all about content delivery -- which is both distributed and highly variable over the day/year. Using Cloud means Netflix can deploy closer to the customer. Plus scale up and down as required. If they did this themselves it would be quite expensive and involve a lot of redundancy.
- closeparen 9y ago
  It’s not either/or. You want to be in a position where your own racks and public cloud providers are all running instances of the same distributed scheduler. Then you can buy capacity wherever it makes the most economic sense at the moment. Just proving that you have the capability to run your own hardware cost-effectively can negotiate Amazon/Google down to that price or lower.
  - nojvek 9y ago
    This is probably the nicest answer. With things like kubernetes, hardware is quite abstracted. If it makes sense to use your own hardware, your software should be good enough to effortlessly shift some load to your own hardware and back incase own hardware can't handle a spike. Hybrids are probably the future where you rent when it makes sense, and buy when it makes sense and move compute and storage by dragging a slider.
assblaster 9y ago
I think long term, Netflix needs redundancy, especially when the primary supplier is a competitor in the same space.
- ocdtrekkie 9y ago
  And the standing problem now is that their secondary supplier they're considering is also a competitor in the same space. It's a wonder Netflix isn't testing the waters against Azure, since Microsoft just closed their video sales offerings down.
  - supernovae 9y ago
    Microsoft pulled out of the music space, they're still selling & renting movies/shows and content services are a core feature of azure. Netflix uses its own POPS for of its content delivery.
    - ocdtrekkie 9y ago
      Ah yes, I mixed that up. :/ Though who rents movies from Microsoft? And they don't have a subscription-based service for it, like both Amazon and Google do.
      - tapoxi 9y ago
        Xbox owners.
      - supernovae 9y ago
        I rent movies and tv shows from Microsoft all the time. They have some great sales.
  - intopieces 9y ago
    Isn't it better to be in bed with your competitor so as to have a solid claim against being declared a monopoly?
skywhopper 9y ago
Netflix has been "testing the waters" at GCP for years. And as their infrastructure grows, relying on a single cloud provider would be a silly limitation to put on themselves. Disaster-recovery/failover is a very natural use case for multi-cloud deployments. Not to mention that for certain workloads and patterns, GCP is cheaper and better. Netflix operates at a scale well beyond what most AWS managed services can really address, and broad cloud patterns like object storage, function-as-a-service, VMs, and networks, can be abstracted away given enough resources, which Netflix surely has. Not to mention that at Netflix's scale they have real leverage over cloud providers and what services they offer. They would be stupid _not_ to be playing AWS and GCP against each other for pricing (on which they already get a huge discount) and features. I'm guessing there's some project getting underway to move/duplicate some subset of resources hosted in AWS on GCP. It's likely a huge project that dwarfs most corporations entire cloud presence, but I'm guessing for Netflix it's a small piece of a giant puzzle, and that the vast majority of their operations are still centered at AWS and will be for years and years to come.
- wiremine 9y ago
  You don't have to be Netflix to get AWS, GCP and MS to play against each on pricing. I've seen a number of my clients build a prototype on each platforms, and then use it to make final decision. Part of that final decision is having the cloud vendors compete on price. I personally haven't seen any massive price breaks, but there definitely is room the margins.
  - Twirrim 9y ago
    For years Dell would let it "leak" out to the press that they were looking at supporting AMD chips. It would happen every year like clockwork. Then after a month you'd hear that Dell and Intel had come up with a new deal to remain Intel exclusive.
    - nandhp 9y ago
      Hasn't Dell offered AMD chips for a long time? Dell currently sells AMD-based laptops, and was doing so back in 2008 as well. For example, this Ryzen laptop: http://www.dell.com/en-us/shop/dell-laptops/new-inspiron-15-5000/spd/inspiron-15-5575-laptop http://www.dell.com/en-us/shop/dell-laptops/new-inspiron-15-...
nodex-alex 9y ago
paywall shit
KirinDave 9y ago
Firstly: not very new. Hasn't Netflix been at this for a long time? Secondly, everyone should be checking. Not only is Google doing a lot of subsidization, but their services are really really good. My current employer ended up being unable to use them but it's I acutely regret every day I'm forced to build out things Google already has or deal with Amazon's frustratingly antiquated ELB infrastructure.
- archgoon 9y ago
  What are the problems you've been running into with ELB?
  - dalyons 9y ago
    No http2 support is increasingly limiting
    - bradhe 9y ago
      ALB has HTTP2 support.
      - mwarkentin 9y ago
        Only at the front end, it downgraded to 1.1 between the ALB and your servers.
        manigandham 9y ago
        All load balancing services do that today. It is rare to get HTTP/2 on both ends unless you run your own software like Envoy.
- joevandyk 9y ago
  I’m sure you are aware but AWS has a newish loaf balancer now: https://docs.aws.amazon.com/elasticloadbalancing/latest/application/introduction.html https://docs.aws.amazon.com/elasticloadbalancing/latest/appl...
bhouston 9y ago
I'm surprised they have thought to move some of their infrastructure in house. The clearly have the scale where this would make sense, but it may be that AWS and GCP provide them with such low special prices that it doesn't make sense...
- rockostrich 9y ago
  Their CDN is in-house [1]. They use cloud services for general computing. [1] https://media.netflix.com/en/company-blog/how-netflix-works-with-isps-around-the-globe-to-deliver-a-great-viewing-experience https://media.netflix.com/en/company-blog/how-netflix-works-...
- diminoten 9y ago
  The problem might just be that it's genuinely incredibly difficult to get infrastructure right at this level, and the cost to move in-house is still too high, even accounting for the savings over the long term.
falcolas 9y ago
This could also be titled "Netflix Ensures its AWS Discounts Continue" AWS sales teams are quite responsive to threats of moving in terms of continuing discounts. I've drawn up migration plans solely to use as leverage against AWS for deep discounts in the past.
- maxxxxx 9y ago
  Just curious: Can you state a range of possible discounts against list price? The stuff I deal with is small fry but I always wonder how much cheaper the big guys get their cloud services compared to us.
  - falcolas 9y ago
    Third hand knowledge - since I was not a part of the negotiations - fair warning. The company I worked for was getting discounts in the 40-50% range. It was the only way they could afford to use AWS, frankly, since it was all static provisioning (i.e. no elasticity in the workload). They were also approaching a petabyte of data stored in S3.
scarface74 9y ago
There are a lot of comments saying that Netflix should run its own datacenter. Adrian Cockcraft helped lead Netflix's effort to move to AWS and now is a VP at AWS https://www.linkedin.com/in/adriancockcroft/ https://www.linkedin.com/in/adriancockcroft/ He did an interview with Software Engineering Radio back in 2014 explaining their motivation. http://www.se-radio.net/2014/12/episode-216-adrian-cockcroft-on-the-modern-cloud-based-platform/ http://www.se-radio.net/2014/12/episode-216-adrian-cockcroft...
- stingraycharles 9y ago
  > Adrian Cockcraft helped lead Netflix's effort to move to AWS and now is a VP at AWS. I always get a bad taste in my mouth when I read things like that. Was it really an objective choice, or was this AWS’ way of saying “thank you” ?
  - scarface74 9y ago
    He led the largest migration from on premises to AWS. Who else would have more experience?
  - exhilaration 9y ago
    No way, if I was CTO of a big company (or, say, whoever at Pentagon is in charge of IT), I would love to speak to the guy that shifted Netflix away from in-house datacenters to "The Cloud".
  - hackbinary 9y ago
    And now dropbox owns those datacentres. ;)
  - illumin8 9y ago
    You're ignoring the fact that he spent a few years at Battery Ventures in between Netflix and AWS. Adrian is a well respected professional in his field, has been around for decades, and nobody holds any negative perceptions of him. He wasn't hired by AWS because he migrated datacenters to the cloud. He was hired by AWS because he created this: https://netflix.github.io/ https://netflix.github.io/ - Netflix's OSS, which has allowed them to attract so much great talent, was a direct result of Adrian's passionate open source advocacy. He's doing the same thing at AWS now, and has already accomplished great things like open sourcing all of the AWS documentation. Disclaimer: work at AWS.
simonebrunozzi 9y ago
My prediction is that by 2020 Netflix will move a significant portion of their IT to Google Cloud. Why? Most of their architecture is microservices / linux containers / serverless, and Google is simply way better positioned there (with Kubernetes as their crown jewel). They will eventually win. Source: no source, just my "expert" opinion (6 years at AWS, 2 at VMware's cloud, currently CEO at a startup).
- andyfleming 9y ago
  AWS is catching up as far as Kubernetes support. I might agree that GCP is better positioned for some of that architecture, but they are still comparable. There are also some advantages to AWS. For example, I think that their RDS product is still more mature than Google Cloud SQL. It's all trade-offs.
  - saidmasoud 9y ago
    +1 regarding RDS. A company I spoke to is moving from GCP to AWS, with their biggest pain point being Cloud SQL is WAY less flexible than RDS. While GCP's k8s offering is better/easier to use, they still have some catching up to do. To drive the second point home, GCP recently released VPC Flow Logs, while AWS has had it for a good while now.
- hinkley 9y ago
  How big doesn’t netflix have to get before it’s in their best interests to build data centers of their own? Toolchains are maturing. They’ve figured out ways to get multiple uses from the same hardware. They shouldn’t be beholden to the rollout or geographical plans of two cloud providers. I don’t even mean build out their entire infrastructure. I think even running your base load on your own hardware makes you understand your cloud situation much more profoundly.
  - treis 9y ago
    They have data centers of their own and distribute all of their own content. AWS powers the UI requests and their back office number crunching stuff. Everything after you push play is done via Netflix's own hardware.
thefourthchime 9y ago
I just got back from NAB and one thing clear was that Google is aggressively trying to get into the broadcast video space with their Cloud Offering.
esturk 9y ago
If more people start using GCP, then in the near future, GCP's prices will go up. In contrast, AWS prices may come down to compete with GCP. And when they reach close parity, things will begin to commoditize and get even cheaper across the board. I see this being a good thing overall as an industry.
- manigandham 9y ago
  Why would GCP prices go up?
flamedoge 9y ago
We might have bigger problems if AWS goes down
greggarious 9y ago
This is off topic, but the mention of my data going onto Google cloud made me wonder: does Netflix share viewing histories? I read their privacy policy but it was so mealy mouthed it's hard to make heads or tails of to a non-lawyer.
- mhh__ 9y ago
  To whom? One would imagine that the viewing data they collect would be one of their advantages over their competitors, making it detrimental to their business to sell it?
- TheIronYuppie 9y ago
  Disclosure: I work at Google Cloud on Kubeflow I can't speak for Netflix's privacy policy, but, if your concern is with Google, we never ever ever ever mix/merge Google customer data with customer data. Like ever ever ever. FWIW, we STRONGLY suggest (just to be doubly sure) that you encrypt at rest and/or offer the option to use Customer Supplied Keys. Encrypt all the things! https://cloud.google.com/security/encryption-at-rest/ https://cloud.google.com/security/encryption-at-rest/
coding123 9y ago
This is one reason you don't want to go with specialized databases or queues. If you used a managed Queue or Database, make sure you can get that same exact Queue or Database on another cloud provider.
manigandham 9y ago
This is natural commodification of compute, especially with such large deployments where there will be a lot of in-house tooling and apps that just need raw compute and storage to run, no matter where it comes from. The rise of containers and Kubernetes makes this even easier now. It'll be interesting to see how the clouds differentiate beyond this as most managed services are still rather underwhelming compared to the hype and are also often beat by Kubernetes deployments if available.
jacksmith21006 9y ago
Really surprised this has not happen earlier.
fishdaemon 9y ago
lol, This a "press release" written by a PR firm employed by google etflix, one of Amazon Web Services’ biggest customers, is expanding its use of Google Cloud, AWS’s biggest rival, according to two people with knowledge of the matter. The move, which hasn’t previously been reported, could portend a shift in the balance of power in the cloud computing market

233 points by raleighm 9y ago

Looks like this is paywalled. Edit: Since this is being downvoted, here is the proof: https://screenshots.firefox.com/bgD6FxWvQS4e0sB2/www.theinformation.com https://screenshots.firefox.com/bgD6FxWvQS4e0sB2/www.theinfo... I have no issue with paywalled-content, it should just be noted that this is, at least for some readers.

petercooper 9y ago
I agree. If putting [video] on video links is a thing here, so should [paywall] or similar since it's even more annoying than videos. However, I'm a subscriber to The Information so here's a share link: https://go.theinformation.com/f2b133baedc57a8c https://go.theinformation.com/f2b133baedc57a8c - I think it requres an email, alas, but it's something if people are desperate to read the story.

blackaspen 9y ago

A lot of this article isn't that surprising -- especially the part where they're keeping "Business Critical" data on GCP. Of course they are! Every company needs a disaster recovery plan and in general shouldn't put their eggs in one basket. I'd be interested to see where the AI/ML frameworks differ though, and what additional value one can extract using GCP versus AWS and at what cost/presumptions.

forgot-my-pw 9y ago

I would think if a business use more than 500 servers, it would make sense to start considering having your own datacenter?

blackaspen 9y ago
It's different for everyone. If your 500 servers are for storage, S3/GCS will beat you in cost. For compute, that might be a different story, but then you still have to consider regionality, networking, etc.
- supernovae 9y ago
  Let's be honest, even in the public cloud there are very few orgs that can survive regional failures and most people solve locality through CDN's for content delivery which means a private datacenter isn't really that different from public cloud for most orgs. Also, when when orgs move to the cloud they don't fire their networking team, they still have deskopts/laptops/servers/security devices/printers and large lans/wans to connect / peer already and these teams could staff up for private datacenter without really adding additional costs that aren't already incurred as part of doing business anyway. Our private datacenter is a few thousand servers leased space in a colo and we have one "datacenter dude" who handles hardware issues and our network team spends 90% of their time still on corporate infra vs actual supports with colo space. One can take some up front engineering/planning costs to scope out a private datacenter for about the same cost as it is to scope/plan VPC's and peering/connectivity/vpn services.
  - blackaspen 9y ago
    I completely agree. I jokingly tell everyone I know using AWS to "not use us-east1" and that they need to use another region not just AZs...and they always laugh and say "you were right" with us-east goes down. At least from my view, there's a difference in backbone versus corporate space networking, but you're right, most everyone won't ever realize any savings from that side of things.
    - jtwaleson 9y ago
      How often does us-east-1 go down? I know of about 4 big outages in the last 7 years but you're making it sound like it happens much more frequently. Is there a list of outages somewhere?
      - supernovae 9y ago
        it's not just about it going down.. if you rely only on us-east they have had multiple times when you couldn't autoscale because there was no more capacity... if you rely on running lean and scale/up/scale down events us-east would have bitten your ass a lot the past year... Just for comparison sake even our private colo has had 0 issues the past decade that caused any interruptions to service
wise0wl 9y ago
You really have to factor in the labor costs of infrastructure and operations into that as well. Running your own private "cloud" isn't cheap. And no matter what you do, you aren't going to get even close to feature parity with the public cloud offerings (or even speed, if you are using virtual-machines). There are a lot of trade offs that you have to weigh in that don't always come down to bare dollars and cents.
- ocdtrekkie 9y ago
  Everything you'd have to pay for, Amazon or Google has to pay for for you... plus profit margin. At Netflix scale, it's almost impossible for someone else's cloud to be cheaper than running your own, and hiring the necessary staff and acquiring the necessary space and hardware to do so. I honestly have to wonder if the level of Amazon's discount for Netflix is absurd to the tune of "we lose money doing this" just to retain them. AWS may need Netflix, to make serving it's smaller clients economically viable, more than Netflix needs AWS.
  - weego 9y ago
    If you're a margin business then certainly. If you're not then there's a lot to be said for keeping your core competency lean and offloading the heavy lifting at a negotiated price.
  - drstewart 9y ago
    This is assuming you can do everything at the same efficiency level as Amazon or Google. It's likely that you wouldn't be able to outperform on these metrics to a degree that it would make the profit margin difference a wash.
    - ocdtrekkie 9y ago
      The exact point I am making is that at a certain point, you are able to do things as efficiently as they are. Netflix certainly qualifies at that scale.
supernovae 9y ago
I wouldn't do my own datacenter as in build out a building, but I would certainly lease a cage and build out my own infrastructure at probably half that count. As far as I know, Netflix has POP's for content delivery in many carrier facilities so it's not a pure cloud player by any means. For the cost of 1 year for a single cloud server you can buy a 52 core dell with 256gb ram and a few SSD's and have price/performance ratios that heavily scale in your favor and with a scheduler like marathon, k8s, nomad you can easily achieve elasticity in a private colo to a large degree without also marrying yourself to cloud specifics and nothing would stop you from peering with major cloud providers for seasonal elastic needs.
- blackaspen 9y ago
  That's a good distinction. CoLo providers are pretty reasonably priced and most of the larger ones are good at what they do. There's a HUGE scale difference between a CoLo and your own -- as soon as you start having to negotiate diesel supply contracts yourself things get crazy fast.
  - supernovae 9y ago
    I would never roll my own unless I was of facebook/google size. Colo space is abundant and affordable.
- willshepherdson 9y ago
  I think this is a crucial distinction. Netflix has a relatively unique architecture for their content delivery system compared to other online video providers, which is worth considering in context of this article. With Netflix's Open Connect (https://openconnect.netflix.com/en/ https://openconnect.netflix.com/en/) system, most residential Internet Service Providers have appliances at the regional point of interconnection. These appliances are inside of or directly connected to the ISP's network in the region. Back in 2016 they had appliances deployed to over 1,000 locations. Content is pre-positioned during off-peak hours to the appliances. Previously Netflix has been open about how they use AWS for "generic, scalable computing" including "all of the logic of the application interface, the content discovery and selection experience, recommendation algorithms, transcoding, etc." At the time, they claimed that one of the advantages of doing this was the ease of use and growing commoditization of the “cloud” market, so it makes sense that they are evaluating alternative cloud providers for these commodity services.
bluejekyll 9y ago
This isn’t such an obvious decision. Running your own datacenter is difficult. Besides racking and stacking, you need to deal with power and cooling. Most likely you will outsource all of this to a collocations management services initially, which becomes your overhead just like AWS/Google. But they have economies of scale, in addition, they have expertise in managing all of this equipment with very high reliability. They also have tons of spare capacity (elastic) that are at your disposal, you’ll lose this in your own datacenter. On top of that you need to hire specialists, and 24/7 noc teams, for making sure your site is always up. And it’s not just one, you need a minimum of two, geographically separated so that they are not effected by the same fire, earthquake, power outage from ice/wind/electrical storms. At the end of the day, You need to beat AWS’ profit margin, and you still would have a huge amount of difficulty being able to match the uptime that you get when you’re so easily able to distribute services across regions and availability zones that are resilient to all the real world problems listed above. And all of those do happen. Even for the largest of companies, I’m not sure it makes sense to compete with these services except in the most niche of markets (like dropbox and storage).
- supernovae 9y ago
  The cloud doesn't really have economies of scale unless your workloads scale opposite of pricing - which they very rarely do. For small shops, sure, but for the example of 500 instances running - you could lease space and grow with physical servers at a better economic scale. If you only need 500 servers for 1 busy season and can drop back down to 100 then stay in the cloud. If you need 500 always running instances and your growth is 20% year over year, i can almost guarantee a good colo will be cheaper and faster if you use the same automation and configuration as code philosophies in private dc as you would in public cloud. Puppet/foreman/cobbler/ansible/chef all work great in private DC
  - bluejekyll 9y ago
    In general, I think it’s a wash. The margins you save on end up costing you in other ways, such as development velocity and having access to easily deployed high-scale solutions like dynamo and managed instances like RDS. it takes time and organizational know-how to get to a point where similar things can be managed in the same way as AWS. Again, there are niche segments where this might not be true, 500 constant machines churning through massive datasets all day-long, etc, might fall into that category, but it’s not slam dunk.
jwilliams 9y ago
It's a good question, but depends on your workload. Netflix is all about content delivery -- which is both distributed and highly variable over the day/year. Using Cloud means Netflix can deploy closer to the customer. Plus scale up and down as required. If they did this themselves it would be quite expensive and involve a lot of redundancy.
closeparen 9y ago
It’s not either/or. You want to be in a position where your own racks and public cloud providers are all running instances of the same distributed scheduler. Then you can buy capacity wherever it makes the most economic sense at the moment. Just proving that you have the capability to run your own hardware cost-effectively can negotiate Amazon/Google down to that price or lower.
- nojvek 9y ago
  This is probably the nicest answer. With things like kubernetes, hardware is quite abstracted. If it makes sense to use your own hardware, your software should be good enough to effortlessly shift some load to your own hardware and back incase own hardware can't handle a spike. Hybrids are probably the future where you rent when it makes sense, and buy when it makes sense and move compute and storage by dragging a slider.

assblaster 9y ago

I think long term, Netflix needs redundancy, especially when the primary supplier is a competitor in the same space.

ocdtrekkie 9y ago
And the standing problem now is that their secondary supplier they're considering is also a competitor in the same space. It's a wonder Netflix isn't testing the waters against Azure, since Microsoft just closed their video sales offerings down.
- supernovae 9y ago
  Microsoft pulled out of the music space, they're still selling & renting movies/shows and content services are a core feature of azure. Netflix uses its own POPS for of its content delivery.
  - ocdtrekkie 9y ago
    Ah yes, I mixed that up. :/ Though who rents movies from Microsoft? And they don't have a subscription-based service for it, like both Amazon and Google do.
    - tapoxi 9y ago
      Xbox owners.
    - supernovae 9y ago
      I rent movies and tv shows from Microsoft all the time. They have some great sales.
- intopieces 9y ago
  Isn't it better to be in bed with your competitor so as to have a solid claim against being declared a monopoly?

skywhopper 9y ago

Netflix has been "testing the waters" at GCP for years. And as their infrastructure grows, relying on a single cloud provider would be a silly limitation to put on themselves. Disaster-recovery/failover is a very natural use case for multi-cloud deployments. Not to mention that for certain workloads and patterns, GCP is cheaper and better. Netflix operates at a scale well beyond what most AWS managed services can really address, and broad cloud patterns like object storage, function-as-a-service, VMs, and networks, can be abstracted away given enough resources, which Netflix surely has. Not to mention that at Netflix's scale they have real leverage over cloud providers and what services they offer. They would be stupid _not_ to be playing AWS and GCP against each other for pricing (on which they already get a huge discount) and features. I'm guessing there's some project getting underway to move/duplicate some subset of resources hosted in AWS on GCP. It's likely a huge project that dwarfs most corporations entire cloud presence, but I'm guessing for Netflix it's a small piece of a giant puzzle, and that the vast majority of their operations are still centered at AWS and will be for years and years to come.

wiremine 9y ago
You don't have to be Netflix to get AWS, GCP and MS to play against each on pricing. I've seen a number of my clients build a prototype on each platforms, and then use it to make final decision. Part of that final decision is having the cloud vendors compete on price. I personally haven't seen any massive price breaks, but there definitely is room the margins.
- Twirrim 9y ago
  For years Dell would let it "leak" out to the press that they were looking at supporting AMD chips. It would happen every year like clockwork. Then after a month you'd hear that Dell and Intel had come up with a new deal to remain Intel exclusive.
  - nandhp 9y ago
    Hasn't Dell offered AMD chips for a long time? Dell currently sells AMD-based laptops, and was doing so back in 2008 as well. For example, this Ryzen laptop: http://www.dell.com/en-us/shop/dell-laptops/new-inspiron-15-5000/spd/inspiron-15-5575-laptop http://www.dell.com/en-us/shop/dell-laptops/new-inspiron-15-...

nodex-alex 9y ago

paywall shit

KirinDave 9y ago

Firstly: not very new. Hasn't Netflix been at this for a long time? Secondly, everyone should be checking. Not only is Google doing a lot of subsidization, but their services are really really good. My current employer ended up being unable to use them but it's I acutely regret every day I'm forced to build out things Google already has or deal with Amazon's frustratingly antiquated ELB infrastructure.

archgoon 9y ago
What are the problems you've been running into with ELB?
- dalyons 9y ago
  No http2 support is increasingly limiting
  - bradhe 9y ago
    ALB has HTTP2 support.
    - mwarkentin 9y ago
      Only at the front end, it downgraded to 1.1 between the ALB and your servers.
      - manigandham 9y ago
        All load balancing services do that today. It is rare to get HTTP/2 on both ends unless you run your own software like Envoy.
joevandyk 9y ago
I’m sure you are aware but AWS has a newish loaf balancer now: https://docs.aws.amazon.com/elasticloadbalancing/latest/application/introduction.html https://docs.aws.amazon.com/elasticloadbalancing/latest/appl...

bhouston 9y ago

I'm surprised they have thought to move some of their infrastructure in house. The clearly have the scale where this would make sense, but it may be that AWS and GCP provide them with such low special prices that it doesn't make sense...

rockostrich 9y ago
Their CDN is in-house [1]. They use cloud services for general computing. [1] https://media.netflix.com/en/company-blog/how-netflix-works-with-isps-around-the-globe-to-deliver-a-great-viewing-experience https://media.netflix.com/en/company-blog/how-netflix-works-...
diminoten 9y ago
The problem might just be that it's genuinely incredibly difficult to get infrastructure right at this level, and the cost to move in-house is still too high, even accounting for the savings over the long term.

falcolas 9y ago

This could also be titled "Netflix Ensures its AWS Discounts Continue" AWS sales teams are quite responsive to threats of moving in terms of continuing discounts. I've drawn up migration plans solely to use as leverage against AWS for deep discounts in the past.

maxxxxx 9y ago
Just curious: Can you state a range of possible discounts against list price? The stuff I deal with is small fry but I always wonder how much cheaper the big guys get their cloud services compared to us.
- falcolas 9y ago
  Third hand knowledge - since I was not a part of the negotiations - fair warning. The company I worked for was getting discounts in the 40-50% range. It was the only way they could afford to use AWS, frankly, since it was all static provisioning (i.e. no elasticity in the workload). They were also approaching a petabyte of data stored in S3.

scarface74 9y ago

There are a lot of comments saying that Netflix should run its own datacenter. Adrian Cockcraft helped lead Netflix's effort to move to AWS and now is a VP at AWS https://www.linkedin.com/in/adriancockcroft/ https://www.linkedin.com/in/adriancockcroft/ He did an interview with Software Engineering Radio back in 2014 explaining their motivation. http://www.se-radio.net/2014/12/episode-216-adrian-cockcroft-on-the-modern-cloud-based-platform/ http://www.se-radio.net/2014/12/episode-216-adrian-cockcroft...

stingraycharles 9y ago
> Adrian Cockcraft helped lead Netflix's effort to move to AWS and now is a VP at AWS. I always get a bad taste in my mouth when I read things like that. Was it really an objective choice, or was this AWS’ way of saying “thank you” ?
- scarface74 9y ago
  He led the largest migration from on premises to AWS. Who else would have more experience?
- exhilaration 9y ago
  No way, if I was CTO of a big company (or, say, whoever at Pentagon is in charge of IT), I would love to speak to the guy that shifted Netflix away from in-house datacenters to "The Cloud".
- hackbinary 9y ago
  And now dropbox owns those datacentres. ;)
- illumin8 9y ago
  You're ignoring the fact that he spent a few years at Battery Ventures in between Netflix and AWS. Adrian is a well respected professional in his field, has been around for decades, and nobody holds any negative perceptions of him. He wasn't hired by AWS because he migrated datacenters to the cloud. He was hired by AWS because he created this: https://netflix.github.io/ https://netflix.github.io/ - Netflix's OSS, which has allowed them to attract so much great talent, was a direct result of Adrian's passionate open source advocacy. He's doing the same thing at AWS now, and has already accomplished great things like open sourcing all of the AWS documentation. Disclaimer: work at AWS.

simonebrunozzi 9y ago

My prediction is that by 2020 Netflix will move a significant portion of their IT to Google Cloud. Why? Most of their architecture is microservices / linux containers / serverless, and Google is simply way better positioned there (with Kubernetes as their crown jewel). They will eventually win. Source: no source, just my "expert" opinion (6 years at AWS, 2 at VMware's cloud, currently CEO at a startup).

andyfleming 9y ago
AWS is catching up as far as Kubernetes support. I might agree that GCP is better positioned for some of that architecture, but they are still comparable. There are also some advantages to AWS. For example, I think that their RDS product is still more mature than Google Cloud SQL. It's all trade-offs.
- saidmasoud 9y ago
  +1 regarding RDS. A company I spoke to is moving from GCP to AWS, with their biggest pain point being Cloud SQL is WAY less flexible than RDS. While GCP's k8s offering is better/easier to use, they still have some catching up to do. To drive the second point home, GCP recently released VPC Flow Logs, while AWS has had it for a good while now.
hinkley 9y ago
How big doesn’t netflix have to get before it’s in their best interests to build data centers of their own? Toolchains are maturing. They’ve figured out ways to get multiple uses from the same hardware. They shouldn’t be beholden to the rollout or geographical plans of two cloud providers. I don’t even mean build out their entire infrastructure. I think even running your base load on your own hardware makes you understand your cloud situation much more profoundly.
- treis 9y ago
  They have data centers of their own and distribute all of their own content. AWS powers the UI requests and their back office number crunching stuff. Everything after you push play is done via Netflix's own hardware.

thefourthchime 9y ago

I just got back from NAB and one thing clear was that Google is aggressively trying to get into the broadcast video space with their Cloud Offering.

esturk 9y ago

If more people start using GCP, then in the near future, GCP's prices will go up. In contrast, AWS prices may come down to compete with GCP. And when they reach close parity, things will begin to commoditize and get even cheaper across the board. I see this being a good thing overall as an industry.

manigandham 9y ago
Why would GCP prices go up?

flamedoge 9y ago

We might have bigger problems if AWS goes down

greggarious 9y ago

This is off topic, but the mention of my data going onto Google cloud made me wonder: does Netflix share viewing histories? I read their privacy policy but it was so mealy mouthed it's hard to make heads or tails of to a non-lawyer.

mhh__ 9y ago
To whom? One would imagine that the viewing data they collect would be one of their advantages over their competitors, making it detrimental to their business to sell it?
TheIronYuppie 9y ago
Disclosure: I work at Google Cloud on Kubeflow I can't speak for Netflix's privacy policy, but, if your concern is with Google, we never ever ever ever mix/merge Google customer data with customer data. Like ever ever ever. FWIW, we STRONGLY suggest (just to be doubly sure) that you encrypt at rest and/or offer the option to use Customer Supplied Keys. Encrypt all the things! https://cloud.google.com/security/encryption-at-rest/ https://cloud.google.com/security/encryption-at-rest/

coding123 9y ago

This is one reason you don't want to go with specialized databases or queues. If you used a managed Queue or Database, make sure you can get that same exact Queue or Database on another cloud provider.

manigandham 9y ago

This is natural commodification of compute, especially with such large deployments where there will be a lot of in-house tooling and apps that just need raw compute and storage to run, no matter where it comes from. The rise of containers and Kubernetes makes this even easier now. It'll be interesting to see how the clouds differentiate beyond this as most managed services are still rather underwhelming compared to the hype and are also often beat by Kubernetes deployments if available.

jacksmith21006 9y ago

Really surprised this has not happen earlier.

fishdaemon 9y ago

lol, This a "press release" written by a PR firm employed by google etflix, one of Amazon Web Services’ biggest customers, is expanding its use of Google Cloud, AWS’s biggest rival, according to two people with knowledge of the matter. The move, which hasn’t previously been reported, could portend a shift in the balance of power in the cloud computing market