6 ms·

This is terrible advice for all but the largest of organizations. Running your own hardware is AWFUL. Get ready to dedicate an entire team to network engineeri

This is terrible advice for all but the largest of organizations. Running your own hardware is AWFUL. Get ready to dedicate an entire team to network engineering, fixing broken hard disks, patching operating systems, screwing around with RAID controllers, upgrading switches, planning power and cooling, and endless vendor negotiation -- with ISPs, hardware manufacturers, datacenter operators, etc. Oh, and did I mention, throw elasticity out the window -- all of this has to be planned in advance, and purchased, and installed, months ahead of when it will be operable. So forget the ease and convenience of just spinning up more capacity. Also, there's a massive distraction of having to focus management attention on this non-value-adding part of the business, all so that you can shave down some cost, rather than investing in growing revenue. Having been in this position firsthand, don't do this. If netflix can run 1/3 of Internet traffic off of AWS, I guarantee they're much larger than you, and it should say something, that they'd rather outsource this part of their business than dealing with all this crap. Focus on software and product/market fit. It's just a much, much, much better use of expensive technical people, that will be done on day 1, without any risk, hassle, or complexity, than trying to replicate something someone else already does for you, much better and at competitive costs, than trying to reinvent all of this yourself. To be honest, I don't even want to deal with EC2 anymore; I'd rather just use a PaaS.

toomuchtodo 8y ago
Netflix runs a third of internet traffic off of their OpenConnect CDN appliances, not AWS (Netflix.com and other control plane-ish functions aside). Once you’re spending a few million dollars a year on cloud (arguably even less in some cases), it behooves you to explore hosting yourself. Cloud provider margins are substantial for a reason.
- eldavido 8y ago
  If you're truly spending "a few million dollars"/year, then MAYBE. 200k/month would get you there. My personal breakpoints are 50k/mo PaaS->EC2/IaaS and then something like 100k or 150k start thinking about physical. Maybe. By the time you hire everyone, get all the planning right, etc. you might be there need-wise, but it's not a sure thing.
  - deleted 8y ago
    [deleted]
  - jjeaff 8y ago
    Why not just rent dedicated bare metal from a large provider?
    - mmt 8y ago
      I'd expect that to have nearly all the disadvantages of cloud (other than virtualization). One is still locked into the provider's pricing structure. One is still locked into the provider's ISP choices. One is still locked into the provider's internal network architecture choices. One is still locked into the provider's limited hardware choices. The last one is the biggest one, if only because there are so many opportunities in so many components to maximize performance and minimize cost, with a little forethought and customization, all while staying well within the commodity market. The "one size fits all" model does many people a disservice. This is especially true even at fairly low scale, where the issue of obtaining certain components in large volumes doesn't come into play. I understand why some people insist it's just too hard to deal with hardware, but I find it disingenuous to advocate that opnion without at least a more comprehensive understanding of what modern, commodity hardware is capable of. I think most of the claims of difficulty are coming from those without this knowledge and have primarily software backgrounds (and are relying on hearsay or other second-hand experience).
      - jjeaff 8y ago
        Most providers I have seen that rent hardware offer unlimited customization when it come to the box you are renting. Network architecture would obviously be off the table though.
Johnny555 8y ago
It's been a few years since doing the cost benefit analysis between AWS and self-hosting, but for around 50 racks worth of servers and storage, the numbers came in on AWS's side. That didn't even take into account the "free" multi-region capability you get from Amazon. Splitting our physical servers into a second region with enough capacity to failover would have nearly doubled our costs.
- jjeaff 8y ago
  Why would you compare AWS vs managing your own data center? You could also compare AWS vs building your own silicon. I think it would be better to compare AWS vs renting dedicated servers from a large provider? I think you will find that the scales tip heavily in favor of renting bare metal as far as price is concerned.
  - Johnny555 8y ago
    Why would you compare AWS vs managing your own data center? Because we were already managing our own data center. I think it would be better to compare AWS vs renting dedicated servers from a large provider? I think you will find that the scales tip heavily in favor of renting bare metal as far as price is concerned. We offloaded a lot of work to Amazon that we were doing ourselves -- database hosting, storage system management, etc (lots of little used data went into S3/Glacier that previously we had on live disks) Also, we liked the ability to have a failover region essentially for free - we only pay for enough servers to replicate the key data we need for failover, and keep the rest of the infrastructure powered off.
    - dominias 8y ago
      You may be managing your own datacenter but you sure won’t be building any new ones.
    - mmt 8y ago
      > storage system management I was a bit incredulous that any truly all-inclusive analysis could ever show AWS being cheaper, but this phrasing made me realize that it could have been the one (remarkably common) case where it usually does: enterprise hardware. That world is easily more expensive than AWS, especially considering that hardware maintenance contracts are a thing (and a shockingly expensive one, to those of us accustomed to the commodity hardware world). > Also, we liked the ability to have a failover region essentially for free - we only pay for enough servers to replicate the key data we need for failover, and keep the rest of the infrastructure powered off. That's a useful advantage, though there's a pitfall in that there's no powering off EBS volumes.
niftich 8y ago
I get what you're saying about the benefits of PaaS or other managed services, but Netflix serves most content (by volume) off of appliances [1] they ship to ISPs that feed into the network at points-of-presence somewhere close to an edge. They use AWS for logic and compute (including transcoding [3]), but video streams from this globally-distributed CDN [2]. [1] https://openconnect.netflix.com/en/ https://openconnect.netflix.com/en/ [2] https://media.netflix.com/en/company-blog/how-netflix-works-with-isps-around-the-globe-to-deliver-a-great-viewing-experience https://media.netflix.com/en/company-blog/how-netflix-works-... [3] https://medium.com/netflix-techblog/high-quality-video-encoding-at-scale-d159db052746 https://medium.com/netflix-techblog/high-quality-video-encod...
deleted 8y ago
[deleted]
patrickg_zill 8y ago
Having run a small hosting company for about 17 years now, I am laughing at this post... You greatly overstated the difficulty and understate the massive cost savings.
- darkerside 8y ago
  When it's your business, of course these things are easy. It's just part of routine maintenance and overhead. When you're in a different business, you generally want to be focusing on your own core competencies and solving your own specific business challenges, not care and feeding of servers. Classic division of labor problem.
- outworlder 8y ago
  No they didn't. Because you are a hosting company. Not all companies have your expertise - most don't.
  - sitkack 8y ago
    The hyperbolic difficulty that folks are applying to running your own hardware is laughable. It is as if by miracle alone that civilization made it through the precarious transitionary period where we racked our own servers into the Wonderous Utopia of Cloud. I definitely wouldn't tie my own shoe laces, only the largest orgs need laces, almost everyone can focus on walking if they use Velcro.
    - tatersolid 8y ago
      It’s about staff and focus. Right now we have 3 sysadmins for our own DC, and they can’t keep up with all the maintenance (upgrades on infra are the worst. Software quality on switches/SANs/appliances is terrifying. And those are better than quality of Server management, BIOS, etc. Even VMware has caused lots of pain with patches and upgrades in recent years. If you don’t ever patch/upgrade to fix all the security vulnerabilities that exist at the infrastructure layer maybe “do it yourself” is cheaper. But that doesn’t fly in the banking industry. We’re half-in the cloud and moving more there just to keep our head above water. We’re spending way less on cloud services than the $500K/year it would cost to double our infrastructure staff and keep it all in house.
hueving 8y ago
>This is terrible advice for all but the largest of organizations. Don't start a conversation with an opening generalization like that if you want something constructive. Especially when the rest of your post is clearly based on the single anecdote of your experience. >Running your own hardware is AWFUL. Maybe for you. Not for any sysadmin with even just a couple of years of experience. >patching operating systems We're talking about instances, none of that sysadmin stuff goes away if you're on AWS. If you don't have patching management for operating systems on AWS then your instances are screwed. AWS instances don't eliminate the need for sysadmin work. The only real difference is the hardware management. And if you read my post you would have seen that I said using aws for the on-demand flexibility is okay. All of the static workloads are what belongs down in your datacenter. Netflix doesn't run 1/3 of the Internet traffic off of AWS, only a tiny subset because of the aforementioned shitty economics. The real workhorses are in custom netflix servers at peering points. Netflix would be bankrupt if they used AWS for video. Do some research before spreading free marketing propaganda. This forum tends to only think in terms of explosive growth of traffic, which <0.1% of companies actually have to deal with. AWS flexibility is needed by very few successful B2C companies, but it's supported by the cargo-culting of orders of magnitude more developers ("jedberg said this worked for reddit, we need this because we're like reddit"). Also, your whole argument about non 'value-add' is bogus. That's the same excuse that management uses to outsource all development. Everything has a cost and provides some value to the company.
- dominias 8y ago
  You got to accept that building and maintaining data enters are simply an old and obsolete model of doing IT that will eventually go away.
  - deleted 8y ago
    [deleted]
- not_kurt_godel 8y ago
  > Don't start a conversation with an opening generalization like that if you want something constructive. Especially when the rest of your post is clearly based on the single anecdote of your experience. You are doing as much, if not more generalization by way of the assumptions you're making. > >Running your own hardware is AWFUL. > Maybe for you. Not for any sysadmin with even just a couple of years of experience. Sysadmins aren't real estate attorneys or facilities managers or security personnel. A lot of them aren't even tech ops who physically manage the DC hardware and are oncall 24/7 to fix problems on-site if needed. You need all of those things to run your own DC, and potentially a lot more if you're physically building the center itself (architects, contractors, civil engineers, etc.). And if you're serious about latency, availability, and durability, you're going to need those things in multiples for however many datacenters are needed to meet your targets. How many organizations have the millions of dollars of capex needed to get that off the ground and keep it all running? > Netflix doesn't run 1/3 of the Internet traffic off of AWS, only a tiny subset because of the aforementioned shitty economics. To how many organizations do the economics of serving 1/3 of Internet traffic apply? 2? How is that a counterexample to his point about datacenters only making sense for the very largest? Even if you sidestep all those costs by renting instead of building and even if we take for granted that your "shitty economics" are still shitty down numerous orders of magnitude from Netflix-size, you're still burning money making your devs design and operate your system twice - once for the DC, once for AWS, and however much work it is to glue the 2 together. The end result may be cheaper infrastructure-wise but it will also be unavoidably less reliable and more complex purely by virtue of having more than twice as many moving parts. Let's say implementing things this way takes a single dev time-and-a-half compared to just doing it on one or the other. Let's say (very conservatively) you pay your dev $100k a year + $50k benefits. You're now $75k in the hole from the get-go. That's enough to pay for roughly 83 m5.large EC2 instances on-demand (no RI) for a year. Your company has to be very large for the marginal savings of using a DC to outweigh that kind of deficit.
deleted 8y ago
[deleted]

toomuchtodo 8y ago

Netflix runs a third of internet traffic off of their OpenConnect CDN appliances, not AWS (Netflix.com and other control plane-ish functions aside). Once you’re spending a few million dollars a year on cloud (arguably even less in some cases), it behooves you to explore hosting yourself. Cloud provider margins are substantial for a reason.

eldavido 8y ago
If you're truly spending "a few million dollars"/year, then MAYBE. 200k/month would get you there. My personal breakpoints are 50k/mo PaaS->EC2/IaaS and then something like 100k or 150k start thinking about physical. Maybe. By the time you hire everyone, get all the planning right, etc. you might be there need-wise, but it's not a sure thing.
- deleted 8y ago
  [deleted]
- jjeaff 8y ago
  Why not just rent dedicated bare metal from a large provider?
  - mmt 8y ago
    I'd expect that to have nearly all the disadvantages of cloud (other than virtualization). One is still locked into the provider's pricing structure. One is still locked into the provider's ISP choices. One is still locked into the provider's internal network architecture choices. One is still locked into the provider's limited hardware choices. The last one is the biggest one, if only because there are so many opportunities in so many components to maximize performance and minimize cost, with a little forethought and customization, all while staying well within the commodity market. The "one size fits all" model does many people a disservice. This is especially true even at fairly low scale, where the issue of obtaining certain components in large volumes doesn't come into play. I understand why some people insist it's just too hard to deal with hardware, but I find it disingenuous to advocate that opnion without at least a more comprehensive understanding of what modern, commodity hardware is capable of. I think most of the claims of difficulty are coming from those without this knowledge and have primarily software backgrounds (and are relying on hearsay or other second-hand experience).
    - jjeaff 8y ago
      Most providers I have seen that rent hardware offer unlimited customization when it come to the box you are renting. Network architecture would obviously be off the table though.

Johnny555 8y ago

It's been a few years since doing the cost benefit analysis between AWS and self-hosting, but for around 50 racks worth of servers and storage, the numbers came in on AWS's side. That didn't even take into account the "free" multi-region capability you get from Amazon. Splitting our physical servers into a second region with enough capacity to failover would have nearly doubled our costs.

jjeaff 8y ago
Why would you compare AWS vs managing your own data center? You could also compare AWS vs building your own silicon. I think it would be better to compare AWS vs renting dedicated servers from a large provider? I think you will find that the scales tip heavily in favor of renting bare metal as far as price is concerned.
- Johnny555 8y ago
  Why would you compare AWS vs managing your own data center? Because we were already managing our own data center. I think it would be better to compare AWS vs renting dedicated servers from a large provider? I think you will find that the scales tip heavily in favor of renting bare metal as far as price is concerned. We offloaded a lot of work to Amazon that we were doing ourselves -- database hosting, storage system management, etc (lots of little used data went into S3/Glacier that previously we had on live disks) Also, we liked the ability to have a failover region essentially for free - we only pay for enough servers to replicate the key data we need for failover, and keep the rest of the infrastructure powered off.
  - dominias 8y ago
    You may be managing your own datacenter but you sure won’t be building any new ones.
  - mmt 8y ago
    > storage system management I was a bit incredulous that any truly all-inclusive analysis could ever show AWS being cheaper, but this phrasing made me realize that it could have been the one (remarkably common) case where it usually does: enterprise hardware. That world is easily more expensive than AWS, especially considering that hardware maintenance contracts are a thing (and a shockingly expensive one, to those of us accustomed to the commodity hardware world). > Also, we liked the ability to have a failover region essentially for free - we only pay for enough servers to replicate the key data we need for failover, and keep the rest of the infrastructure powered off. That's a useful advantage, though there's a pitfall in that there's no powering off EBS volumes.

niftich 8y ago

I get what you're saying about the benefits of PaaS or other managed services, but Netflix serves most content (by volume) off of appliances [1] they ship to ISPs that feed into the network at points-of-presence somewhere close to an edge. They use AWS for logic and compute (including transcoding [3]), but video streams from this globally-distributed CDN [2]. [1] https://openconnect.netflix.com/en/ https://openconnect.netflix.com/en/ [2] https://media.netflix.com/en/company-blog/how-netflix-works-with-isps-around-the-globe-to-deliver-a-great-viewing-experience https://media.netflix.com/en/company-blog/how-netflix-works-... [3] https://medium.com/netflix-techblog/high-quality-video-encoding-at-scale-d159db052746 https://medium.com/netflix-techblog/high-quality-video-encod...

deleted 8y ago

[deleted]

patrickg_zill 8y ago

Having run a small hosting company for about 17 years now, I am laughing at this post... You greatly overstated the difficulty and understate the massive cost savings.

darkerside 8y ago
When it's your business, of course these things are easy. It's just part of routine maintenance and overhead. When you're in a different business, you generally want to be focusing on your own core competencies and solving your own specific business challenges, not care and feeding of servers. Classic division of labor problem.
outworlder 8y ago
No they didn't. Because you are a hosting company. Not all companies have your expertise - most don't.
- sitkack 8y ago
  The hyperbolic difficulty that folks are applying to running your own hardware is laughable. It is as if by miracle alone that civilization made it through the precarious transitionary period where we racked our own servers into the Wonderous Utopia of Cloud. I definitely wouldn't tie my own shoe laces, only the largest orgs need laces, almost everyone can focus on walking if they use Velcro.
  - tatersolid 8y ago
    It’s about staff and focus. Right now we have 3 sysadmins for our own DC, and they can’t keep up with all the maintenance (upgrades on infra are the worst. Software quality on switches/SANs/appliances is terrifying. And those are better than quality of Server management, BIOS, etc. Even VMware has caused lots of pain with patches and upgrades in recent years. If you don’t ever patch/upgrade to fix all the security vulnerabilities that exist at the infrastructure layer maybe “do it yourself” is cheaper. But that doesn’t fly in the banking industry. We’re half-in the cloud and moving more there just to keep our head above water. We’re spending way less on cloud services than the $500K/year it would cost to double our infrastructure staff and keep it all in house.

hueving 8y ago

>This is terrible advice for all but the largest of organizations. Don't start a conversation with an opening generalization like that if you want something constructive. Especially when the rest of your post is clearly based on the single anecdote of your experience. >Running your own hardware is AWFUL. Maybe for you. Not for any sysadmin with even just a couple of years of experience. >patching operating systems We're talking about instances, none of that sysadmin stuff goes away if you're on AWS. If you don't have patching management for operating systems on AWS then your instances are screwed. AWS instances don't eliminate the need for sysadmin work. The only real difference is the hardware management. And if you read my post you would have seen that I said using aws for the on-demand flexibility is okay. All of the static workloads are what belongs down in your datacenter. Netflix doesn't run 1/3 of the Internet traffic off of AWS, only a tiny subset because of the aforementioned shitty economics. The real workhorses are in custom netflix servers at peering points. Netflix would be bankrupt if they used AWS for video. Do some research before spreading free marketing propaganda. This forum tends to only think in terms of explosive growth of traffic, which <0.1% of companies actually have to deal with. AWS flexibility is needed by very few successful B2C companies, but it's supported by the cargo-culting of orders of magnitude more developers ("jedberg said this worked for reddit, we need this because we're like reddit"). Also, your whole argument about non 'value-add' is bogus. That's the same excuse that management uses to outsource all development. Everything has a cost and provides some value to the company.

dominias 8y ago
You got to accept that building and maintaining data enters are simply an old and obsolete model of doing IT that will eventually go away.
- deleted 8y ago
  [deleted]
not_kurt_godel 8y ago
> Don't start a conversation with an opening generalization like that if you want something constructive. Especially when the rest of your post is clearly based on the single anecdote of your experience. You are doing as much, if not more generalization by way of the assumptions you're making. > >Running your own hardware is AWFUL. > Maybe for you. Not for any sysadmin with even just a couple of years of experience. Sysadmins aren't real estate attorneys or facilities managers or security personnel. A lot of them aren't even tech ops who physically manage the DC hardware and are oncall 24/7 to fix problems on-site if needed. You need all of those things to run your own DC, and potentially a lot more if you're physically building the center itself (architects, contractors, civil engineers, etc.). And if you're serious about latency, availability, and durability, you're going to need those things in multiples for however many datacenters are needed to meet your targets. How many organizations have the millions of dollars of capex needed to get that off the ground and keep it all running? > Netflix doesn't run 1/3 of the Internet traffic off of AWS, only a tiny subset because of the aforementioned shitty economics. To how many organizations do the economics of serving 1/3 of Internet traffic apply? 2? How is that a counterexample to his point about datacenters only making sense for the very largest? Even if you sidestep all those costs by renting instead of building and even if we take for granted that your "shitty economics" are still shitty down numerous orders of magnitude from Netflix-size, you're still burning money making your devs design and operate your system twice - once for the DC, once for AWS, and however much work it is to glue the 2 together. The end result may be cheaper infrastructure-wise but it will also be unavoidably less reliable and more complex purely by virtue of having more than twice as many moving parts. Let's say implementing things this way takes a single dev time-and-a-half compared to just doing it on one or the other. Let's say (very conservatively) you pay your dev $100k a year + $50k benefits. You're now $75k in the hole from the get-go. That's enough to pay for roughly 83 m5.large EC2 instances on-demand (no RI) for a year. Your company has to be very large for the marginal savings of using a DC to outweigh that kind of deficit.

deleted 8y ago

[deleted]