5 ms·

I would think if a business use more than 500 servers, it would make sense to start considering having your own datacenter?

blackaspen 9y ago
It's different for everyone. If your 500 servers are for storage, S3/GCS will beat you in cost. For compute, that might be a different story, but then you still have to consider regionality, networking, etc.
- supernovae 9y ago
  Let's be honest, even in the public cloud there are very few orgs that can survive regional failures and most people solve locality through CDN's for content delivery which means a private datacenter isn't really that different from public cloud for most orgs. Also, when when orgs move to the cloud they don't fire their networking team, they still have deskopts/laptops/servers/security devices/printers and large lans/wans to connect / peer already and these teams could staff up for private datacenter without really adding additional costs that aren't already incurred as part of doing business anyway. Our private datacenter is a few thousand servers leased space in a colo and we have one "datacenter dude" who handles hardware issues and our network team spends 90% of their time still on corporate infra vs actual supports with colo space. One can take some up front engineering/planning costs to scope out a private datacenter for about the same cost as it is to scope/plan VPC's and peering/connectivity/vpn services.
  - blackaspen 9y ago
    I completely agree. I jokingly tell everyone I know using AWS to "not use us-east1" and that they need to use another region not just AZs...and they always laugh and say "you were right" with us-east goes down. At least from my view, there's a difference in backbone versus corporate space networking, but you're right, most everyone won't ever realize any savings from that side of things.
    - jtwaleson 9y ago
      How often does us-east-1 go down? I know of about 4 big outages in the last 7 years but you're making it sound like it happens much more frequently. Is there a list of outages somewhere?
      - supernovae 9y ago
        it's not just about it going down.. if you rely only on us-east they have had multiple times when you couldn't autoscale because there was no more capacity... if you rely on running lean and scale/up/scale down events us-east would have bitten your ass a lot the past year... Just for comparison sake even our private colo has had 0 issues the past decade that caused any interruptions to service
        risaacs99 9y ago
        I do run such a service, one that must scale daily in response to traffic at multiples of off peak traffic and I can state unequivocally that this is false. I can vaguely recall over the past several years single availability zones having short periods of time where single instance types were unavailable, but never the entire region... The idea that this happens "a lot in the past year" for the entire region isn't my experience at all.
wise0wl 9y ago
You really have to factor in the labor costs of infrastructure and operations into that as well. Running your own private "cloud" isn't cheap. And no matter what you do, you aren't going to get even close to feature parity with the public cloud offerings (or even speed, if you are using virtual-machines). There are a lot of trade offs that you have to weigh in that don't always come down to bare dollars and cents.
- ocdtrekkie 9y ago
  Everything you'd have to pay for, Amazon or Google has to pay for for you... plus profit margin. At Netflix scale, it's almost impossible for someone else's cloud to be cheaper than running your own, and hiring the necessary staff and acquiring the necessary space and hardware to do so. I honestly have to wonder if the level of Amazon's discount for Netflix is absurd to the tune of "we lose money doing this" just to retain them. AWS may need Netflix, to make serving it's smaller clients economically viable, more than Netflix needs AWS.
  - weego 9y ago
    If you're a margin business then certainly. If you're not then there's a lot to be said for keeping your core competency lean and offloading the heavy lifting at a negotiated price.
  - drstewart 9y ago
    This is assuming you can do everything at the same efficiency level as Amazon or Google. It's likely that you wouldn't be able to outperform on these metrics to a degree that it would make the profit margin difference a wash.
    - ocdtrekkie 9y ago
      The exact point I am making is that at a certain point, you are able to do things as efficiently as they are. Netflix certainly qualifies at that scale.
supernovae 9y ago
I wouldn't do my own datacenter as in build out a building, but I would certainly lease a cage and build out my own infrastructure at probably half that count. As far as I know, Netflix has POP's for content delivery in many carrier facilities so it's not a pure cloud player by any means. For the cost of 1 year for a single cloud server you can buy a 52 core dell with 256gb ram and a few SSD's and have price/performance ratios that heavily scale in your favor and with a scheduler like marathon, k8s, nomad you can easily achieve elasticity in a private colo to a large degree without also marrying yourself to cloud specifics and nothing would stop you from peering with major cloud providers for seasonal elastic needs.
- blackaspen 9y ago
  That's a good distinction. CoLo providers are pretty reasonably priced and most of the larger ones are good at what they do. There's a HUGE scale difference between a CoLo and your own -- as soon as you start having to negotiate diesel supply contracts yourself things get crazy fast.
  - supernovae 9y ago
    I would never roll my own unless I was of facebook/google size. Colo space is abundant and affordable.
- willshepherdson 9y ago
  I think this is a crucial distinction. Netflix has a relatively unique architecture for their content delivery system compared to other online video providers, which is worth considering in context of this article. With Netflix's Open Connect (https://openconnect.netflix.com/en/ https://openconnect.netflix.com/en/) system, most residential Internet Service Providers have appliances at the regional point of interconnection. These appliances are inside of or directly connected to the ISP's network in the region. Back in 2016 they had appliances deployed to over 1,000 locations. Content is pre-positioned during off-peak hours to the appliances. Previously Netflix has been open about how they use AWS for "generic, scalable computing" including "all of the logic of the application interface, the content discovery and selection experience, recommendation algorithms, transcoding, etc." At the time, they claimed that one of the advantages of doing this was the ease of use and growing commoditization of the “cloud” market, so it makes sense that they are evaluating alternative cloud providers for these commodity services.
bluejekyll 9y ago
This isn’t such an obvious decision. Running your own datacenter is difficult. Besides racking and stacking, you need to deal with power and cooling. Most likely you will outsource all of this to a collocations management services initially, which becomes your overhead just like AWS/Google. But they have economies of scale, in addition, they have expertise in managing all of this equipment with very high reliability. They also have tons of spare capacity (elastic) that are at your disposal, you’ll lose this in your own datacenter. On top of that you need to hire specialists, and 24/7 noc teams, for making sure your site is always up. And it’s not just one, you need a minimum of two, geographically separated so that they are not effected by the same fire, earthquake, power outage from ice/wind/electrical storms. At the end of the day, You need to beat AWS’ profit margin, and you still would have a huge amount of difficulty being able to match the uptime that you get when you’re so easily able to distribute services across regions and availability zones that are resilient to all the real world problems listed above. And all of those do happen. Even for the largest of companies, I’m not sure it makes sense to compete with these services except in the most niche of markets (like dropbox and storage).
- supernovae 9y ago
  The cloud doesn't really have economies of scale unless your workloads scale opposite of pricing - which they very rarely do. For small shops, sure, but for the example of 500 instances running - you could lease space and grow with physical servers at a better economic scale. If you only need 500 servers for 1 busy season and can drop back down to 100 then stay in the cloud. If you need 500 always running instances and your growth is 20% year over year, i can almost guarantee a good colo will be cheaper and faster if you use the same automation and configuration as code philosophies in private dc as you would in public cloud. Puppet/foreman/cobbler/ansible/chef all work great in private DC
  - bluejekyll 9y ago
    In general, I think it’s a wash. The margins you save on end up costing you in other ways, such as development velocity and having access to easily deployed high-scale solutions like dynamo and managed instances like RDS. it takes time and organizational know-how to get to a point where similar things can be managed in the same way as AWS. Again, there are niche segments where this might not be true, 500 constant machines churning through massive datasets all day-long, etc, might fall into that category, but it’s not slam dunk.
jwilliams 9y ago
It's a good question, but depends on your workload. Netflix is all about content delivery -- which is both distributed and highly variable over the day/year. Using Cloud means Netflix can deploy closer to the customer. Plus scale up and down as required. If they did this themselves it would be quite expensive and involve a lot of redundancy.
closeparen 9y ago
It’s not either/or. You want to be in a position where your own racks and public cloud providers are all running instances of the same distributed scheduler. Then you can buy capacity wherever it makes the most economic sense at the moment. Just proving that you have the capability to run your own hardware cost-effectively can negotiate Amazon/Google down to that price or lower.
- nojvek 9y ago
  This is probably the nicest answer. With things like kubernetes, hardware is quite abstracted. If it makes sense to use your own hardware, your software should be good enough to effortlessly shift some load to your own hardware and back incase own hardware can't handle a spike. Hybrids are probably the future where you rent when it makes sense, and buy when it makes sense and move compute and storage by dragging a slider.

blackaspen 9y ago

It's different for everyone. If your 500 servers are for storage, S3/GCS will beat you in cost. For compute, that might be a different story, but then you still have to consider regionality, networking, etc.

supernovae 9y ago
Let's be honest, even in the public cloud there are very few orgs that can survive regional failures and most people solve locality through CDN's for content delivery which means a private datacenter isn't really that different from public cloud for most orgs. Also, when when orgs move to the cloud they don't fire their networking team, they still have deskopts/laptops/servers/security devices/printers and large lans/wans to connect / peer already and these teams could staff up for private datacenter without really adding additional costs that aren't already incurred as part of doing business anyway. Our private datacenter is a few thousand servers leased space in a colo and we have one "datacenter dude" who handles hardware issues and our network team spends 90% of their time still on corporate infra vs actual supports with colo space. One can take some up front engineering/planning costs to scope out a private datacenter for about the same cost as it is to scope/plan VPC's and peering/connectivity/vpn services.
- blackaspen 9y ago
  I completely agree. I jokingly tell everyone I know using AWS to "not use us-east1" and that they need to use another region not just AZs...and they always laugh and say "you were right" with us-east goes down. At least from my view, there's a difference in backbone versus corporate space networking, but you're right, most everyone won't ever realize any savings from that side of things.
  - jtwaleson 9y ago
    How often does us-east-1 go down? I know of about 4 big outages in the last 7 years but you're making it sound like it happens much more frequently. Is there a list of outages somewhere?
    - supernovae 9y ago
      it's not just about it going down.. if you rely only on us-east they have had multiple times when you couldn't autoscale because there was no more capacity... if you rely on running lean and scale/up/scale down events us-east would have bitten your ass a lot the past year... Just for comparison sake even our private colo has had 0 issues the past decade that caused any interruptions to service
      - risaacs99 9y ago
        I do run such a service, one that must scale daily in response to traffic at multiples of off peak traffic and I can state unequivocally that this is false. I can vaguely recall over the past several years single availability zones having short periods of time where single instance types were unavailable, but never the entire region... The idea that this happens "a lot in the past year" for the entire region isn't my experience at all.

wise0wl 9y ago

You really have to factor in the labor costs of infrastructure and operations into that as well. Running your own private "cloud" isn't cheap. And no matter what you do, you aren't going to get even close to feature parity with the public cloud offerings (or even speed, if you are using virtual-machines). There are a lot of trade offs that you have to weigh in that don't always come down to bare dollars and cents.

ocdtrekkie 9y ago
Everything you'd have to pay for, Amazon or Google has to pay for for you... plus profit margin. At Netflix scale, it's almost impossible for someone else's cloud to be cheaper than running your own, and hiring the necessary staff and acquiring the necessary space and hardware to do so. I honestly have to wonder if the level of Amazon's discount for Netflix is absurd to the tune of "we lose money doing this" just to retain them. AWS may need Netflix, to make serving it's smaller clients economically viable, more than Netflix needs AWS.
- weego 9y ago
  If you're a margin business then certainly. If you're not then there's a lot to be said for keeping your core competency lean and offloading the heavy lifting at a negotiated price.
- drstewart 9y ago
  This is assuming you can do everything at the same efficiency level as Amazon or Google. It's likely that you wouldn't be able to outperform on these metrics to a degree that it would make the profit margin difference a wash.
  - ocdtrekkie 9y ago
    The exact point I am making is that at a certain point, you are able to do things as efficiently as they are. Netflix certainly qualifies at that scale.

supernovae 9y ago

I wouldn't do my own datacenter as in build out a building, but I would certainly lease a cage and build out my own infrastructure at probably half that count. As far as I know, Netflix has POP's for content delivery in many carrier facilities so it's not a pure cloud player by any means. For the cost of 1 year for a single cloud server you can buy a 52 core dell with 256gb ram and a few SSD's and have price/performance ratios that heavily scale in your favor and with a scheduler like marathon, k8s, nomad you can easily achieve elasticity in a private colo to a large degree without also marrying yourself to cloud specifics and nothing would stop you from peering with major cloud providers for seasonal elastic needs.

blackaspen 9y ago
That's a good distinction. CoLo providers are pretty reasonably priced and most of the larger ones are good at what they do. There's a HUGE scale difference between a CoLo and your own -- as soon as you start having to negotiate diesel supply contracts yourself things get crazy fast.
- supernovae 9y ago
  I would never roll my own unless I was of facebook/google size. Colo space is abundant and affordable.
willshepherdson 9y ago
I think this is a crucial distinction. Netflix has a relatively unique architecture for their content delivery system compared to other online video providers, which is worth considering in context of this article. With Netflix's Open Connect (https://openconnect.netflix.com/en/ https://openconnect.netflix.com/en/) system, most residential Internet Service Providers have appliances at the regional point of interconnection. These appliances are inside of or directly connected to the ISP's network in the region. Back in 2016 they had appliances deployed to over 1,000 locations. Content is pre-positioned during off-peak hours to the appliances. Previously Netflix has been open about how they use AWS for "generic, scalable computing" including "all of the logic of the application interface, the content discovery and selection experience, recommendation algorithms, transcoding, etc." At the time, they claimed that one of the advantages of doing this was the ease of use and growing commoditization of the “cloud” market, so it makes sense that they are evaluating alternative cloud providers for these commodity services.

bluejekyll 9y ago

This isn’t such an obvious decision. Running your own datacenter is difficult. Besides racking and stacking, you need to deal with power and cooling. Most likely you will outsource all of this to a collocations management services initially, which becomes your overhead just like AWS/Google. But they have economies of scale, in addition, they have expertise in managing all of this equipment with very high reliability. They also have tons of spare capacity (elastic) that are at your disposal, you’ll lose this in your own datacenter. On top of that you need to hire specialists, and 24/7 noc teams, for making sure your site is always up. And it’s not just one, you need a minimum of two, geographically separated so that they are not effected by the same fire, earthquake, power outage from ice/wind/electrical storms. At the end of the day, You need to beat AWS’ profit margin, and you still would have a huge amount of difficulty being able to match the uptime that you get when you’re so easily able to distribute services across regions and availability zones that are resilient to all the real world problems listed above. And all of those do happen. Even for the largest of companies, I’m not sure it makes sense to compete with these services except in the most niche of markets (like dropbox and storage).

supernovae 9y ago
The cloud doesn't really have economies of scale unless your workloads scale opposite of pricing - which they very rarely do. For small shops, sure, but for the example of 500 instances running - you could lease space and grow with physical servers at a better economic scale. If you only need 500 servers for 1 busy season and can drop back down to 100 then stay in the cloud. If you need 500 always running instances and your growth is 20% year over year, i can almost guarantee a good colo will be cheaper and faster if you use the same automation and configuration as code philosophies in private dc as you would in public cloud. Puppet/foreman/cobbler/ansible/chef all work great in private DC
- bluejekyll 9y ago
  In general, I think it’s a wash. The margins you save on end up costing you in other ways, such as development velocity and having access to easily deployed high-scale solutions like dynamo and managed instances like RDS. it takes time and organizational know-how to get to a point where similar things can be managed in the same way as AWS. Again, there are niche segments where this might not be true, 500 constant machines churning through massive datasets all day-long, etc, might fall into that category, but it’s not slam dunk.

jwilliams 9y ago

It's a good question, but depends on your workload. Netflix is all about content delivery -- which is both distributed and highly variable over the day/year. Using Cloud means Netflix can deploy closer to the customer. Plus scale up and down as required. If they did this themselves it would be quite expensive and involve a lot of redundancy.

closeparen 9y ago

It’s not either/or. You want to be in a position where your own racks and public cloud providers are all running instances of the same distributed scheduler. Then you can buy capacity wherever it makes the most economic sense at the moment. Just proving that you have the capability to run your own hardware cost-effectively can negotiate Amazon/Google down to that price or lower.

nojvek 9y ago
This is probably the nicest answer. With things like kubernetes, hardware is quite abstracted. If it makes sense to use your own hardware, your software should be good enough to effortlessly shift some load to your own hardware and back incase own hardware can't handle a spike. Hybrids are probably the future where you rent when it makes sense, and buy when it makes sense and move compute and storage by dragging a slider.