8 ms·

> Not at this rate. Source? Has there ever been an industry wide survey that compares availability from "insert average colo/data center operations" with the c

by mbesto 5y ago

> Not at this rate. Source? Has there ever been an industry wide survey that compares availability from "insert average colo/data center operations" with the cloud ones? And I'm not talking about "we have 12 SREs who are based in Cupertino and are all paid top dollar to support a colo"...I'm talking average.

jandrewrogers 5y ago
I've worked at a few small companies over the years that had their own significant colos and/or data centers built on the cheap, and only a sysadmin or two to run them. Anecdotally, if the infrastructure is setup right, outages are very rare. Some of these were serving up massive loads at the time. I've done these build-outs a few times in my career and it isn't that hard to do, reliable software is more likely to be an issue. The only significant outage I remember is when the redundant power systems in one DC both failed at the same time for different reasons, which can happen. It is as if the software industry has collectively forgotten how to run basic data center operations. Something that used to be a blue collar skill is now treated like arcane magic.
- ctvo 5y ago
  > It is as if the software industry has collectively forgotten how to run basic data center operations. Something that used to be a blue collar skill is now treated like arcane magic. It's not arcane magic. It's undifferentiated toil that requires hiring for a different skill set than tech companies generally want to hire for. Of course when you get to a certain size it may make sense to take on this cost. I want us to stop pretending individual actors lack the agency to make their own decisions, and they're all blind to how AWS is charging them a fortune for such simple things they can do themselves. You get value from AWS or you stop using AWS.
Spooky23 5y ago
Running a multi-tenant datacenter or hyper scale cloud datacenter is a different business than running a datacenter. The myth of the cost of running facilities on HN is insane - it’s like saying you can’t drive a car unless you hire a formula 1 driver. I worked through the ranks at a large enterprise that ran a “big” datacenter for a decade. The facilities team was about 6 people, average salary around $90k. I can only remember one power interruption affecting more than a rack, caused by a failure during a maintenance event that requires a shutdown for safety reasons. The rest is like any other industrial facility - you have service contracts for the equipment, etc and maintain things. There’s a cost/capability curve that you need to plan around for these matters. You need to make business and engineering decisions based on your actual circumstances. If the answer is automatically “AWS <whatever>“, you’re making a decision to burn dollars for convenience.
- mbesto 5y ago
  > The facilities team was about 6 people, average salary around $90k. Ok so $540k salaries + benefits, so ~$700k. Then you have transaction costs: - Annual salary increases - Any cost associated with people leaving (severance, hiring, recruiters, HR, HR systems) - Systems that run in the data center (logging, monitoring, etc.) - Procurement costs with changing costs in hardware (silicon shortages, etc.) - Security compliance overhead and associated risks - Finance resources required to capitalize and manage asset allocation - etc. etc. Versus - Click a button and voila it works. - Hire way less engineers to manage the system administrative portion > If the answer is automatically “AWS <whatever>“, you’re making a decision to burn dollars for convenience. 100% AGREE. The answer is always "it depends", but just like people are saying "just put in the cloud", the opposite of "well it worked for us using a data center" isn't that simple.
  - Spooky23 5y ago
    Say all of those costs are $2,000,000, and you have 25,000 billable endpoints in the datacenter… you’re looking at less than $0.01/hour for that overhead on a unit basis. Obviously, there’s a huge capital investment component too that has to be incorporated. Those costs may be really high if you’re in a growth phase as you need to overbuy capacity. Just to be clear, I’m not arguing that on-prem is magically cheap. :) But it has its place too!
    - mbesto 5y ago
      Agreed on all accounts.
- f6v 5y ago
  That’s not what parent was asking about. The question is if a company of certain size is more likely to suffer from an outage on AWS compared to own hardware. I’ve been deploying to AWS for years and can’t remember and outage on their side in my region. But this is anecdotal and doesn’t necessarily reflect the statistics.

by mbesto 5y ago

jandrewrogers 5y ago

I've worked at a few small companies over the years that had their own significant colos and/or data centers built on the cheap, and only a sysadmin or two to run them. Anecdotally, if the infrastructure is setup right, outages are very rare. Some of these were serving up massive loads at the time. I've done these build-outs a few times in my career and it isn't that hard to do, reliable software is more likely to be an issue. The only significant outage I remember is when the redundant power systems in one DC both failed at the same time for different reasons, which can happen. It is as if the software industry has collectively forgotten how to run basic data center operations. Something that used to be a blue collar skill is now treated like arcane magic.

ctvo 5y ago
> It is as if the software industry has collectively forgotten how to run basic data center operations. Something that used to be a blue collar skill is now treated like arcane magic. It's not arcane magic. It's undifferentiated toil that requires hiring for a different skill set than tech companies generally want to hire for. Of course when you get to a certain size it may make sense to take on this cost. I want us to stop pretending individual actors lack the agency to make their own decisions, and they're all blind to how AWS is charging them a fortune for such simple things they can do themselves. You get value from AWS or you stop using AWS.

Spooky23 5y ago

Running a multi-tenant datacenter or hyper scale cloud datacenter is a different business than running a datacenter. The myth of the cost of running facilities on HN is insane - it’s like saying you can’t drive a car unless you hire a formula 1 driver. I worked through the ranks at a large enterprise that ran a “big” datacenter for a decade. The facilities team was about 6 people, average salary around $90k. I can only remember one power interruption affecting more than a rack, caused by a failure during a maintenance event that requires a shutdown for safety reasons. The rest is like any other industrial facility - you have service contracts for the equipment, etc and maintain things. There’s a cost/capability curve that you need to plan around for these matters. You need to make business and engineering decisions based on your actual circumstances. If the answer is automatically “AWS <whatever>“, you’re making a decision to burn dollars for convenience.

mbesto 5y ago
> The facilities team was about 6 people, average salary around $90k. Ok so $540k salaries + benefits, so ~$700k. Then you have transaction costs: - Annual salary increases - Any cost associated with people leaving (severance, hiring, recruiters, HR, HR systems) - Systems that run in the data center (logging, monitoring, etc.) - Procurement costs with changing costs in hardware (silicon shortages, etc.) - Security compliance overhead and associated risks - Finance resources required to capitalize and manage asset allocation - etc. etc. Versus - Click a button and voila it works. - Hire way less engineers to manage the system administrative portion > If the answer is automatically “AWS <whatever>“, you’re making a decision to burn dollars for convenience. 100% AGREE. The answer is always "it depends", but just like people are saying "just put in the cloud", the opposite of "well it worked for us using a data center" isn't that simple.
- Spooky23 5y ago
  Say all of those costs are $2,000,000, and you have 25,000 billable endpoints in the datacenter… you’re looking at less than $0.01/hour for that overhead on a unit basis. Obviously, there’s a huge capital investment component too that has to be incorporated. Those costs may be really high if you’re in a growth phase as you need to overbuy capacity. Just to be clear, I’m not arguing that on-prem is magically cheap. :) But it has its place too!
  - mbesto 5y ago
    Agreed on all accounts.
f6v 5y ago
That’s not what parent was asking about. The question is if a company of certain size is more likely to suffer from an outage on AWS compared to own hardware. I’ve been deploying to AWS for years and can’t remember and outage on their side in my region. But this is anecdotal and doesn’t necessarily reflect the statistics.