5 ms·

Self-managed infrastructure doesn’t fail now?

by f6v 5y ago

iso1631 5y ago
Not at this rate. I remember we had a power outage in 2006, it actually took one of my services off air. Since then of course that has been rectified, and the loss of a building wouldn't impact on any of the critical, essential or important services I provide.
- mbesto 5y ago
  > Not at this rate. Source? Has there ever been an industry wide survey that compares availability from "insert average colo/data center operations" with the cloud ones? And I'm not talking about "we have 12 SREs who are based in Cupertino and are all paid top dollar to support a colo"...I'm talking average.
  - jandrewrogers 5y ago
    I've worked at a few small companies over the years that had their own significant colos and/or data centers built on the cheap, and only a sysadmin or two to run them. Anecdotally, if the infrastructure is setup right, outages are very rare. Some of these were serving up massive loads at the time. I've done these build-outs a few times in my career and it isn't that hard to do, reliable software is more likely to be an issue. The only significant outage I remember is when the redundant power systems in one DC both failed at the same time for different reasons, which can happen. It is as if the software industry has collectively forgotten how to run basic data center operations. Something that used to be a blue collar skill is now treated like arcane magic.
    - ctvo 5y ago
      > It is as if the software industry has collectively forgotten how to run basic data center operations. Something that used to be a blue collar skill is now treated like arcane magic. It's not arcane magic. It's undifferentiated toil that requires hiring for a different skill set than tech companies generally want to hire for. Of course when you get to a certain size it may make sense to take on this cost. I want us to stop pretending individual actors lack the agency to make their own decisions, and they're all blind to how AWS is charging them a fortune for such simple things they can do themselves. You get value from AWS or you stop using AWS.
  - Spooky23 5y ago
    Running a multi-tenant datacenter or hyper scale cloud datacenter is a different business than running a datacenter. The myth of the cost of running facilities on HN is insane - it’s like saying you can’t drive a car unless you hire a formula 1 driver. I worked through the ranks at a large enterprise that ran a “big” datacenter for a decade. The facilities team was about 6 people, average salary around $90k. I can only remember one power interruption affecting more than a rack, caused by a failure during a maintenance event that requires a shutdown for safety reasons. The rest is like any other industrial facility - you have service contracts for the equipment, etc and maintain things. There’s a cost/capability curve that you need to plan around for these matters. You need to make business and engineering decisions based on your actual circumstances. If the answer is automatically “AWS <whatever>“, you’re making a decision to burn dollars for convenience.
    - mbesto 5y ago
      > The facilities team was about 6 people, average salary around $90k. Ok so $540k salaries + benefits, so ~$700k. Then you have transaction costs: - Annual salary increases - Any cost associated with people leaving (severance, hiring, recruiters, HR, HR systems) - Systems that run in the data center (logging, monitoring, etc.) - Procurement costs with changing costs in hardware (silicon shortages, etc.) - Security compliance overhead and associated risks - Finance resources required to capitalize and manage asset allocation - etc. etc. Versus - Click a button and voila it works. - Hire way less engineers to manage the system administrative portion > If the answer is automatically “AWS <whatever>“, you’re making a decision to burn dollars for convenience. 100% AGREE. The answer is always "it depends", but just like people are saying "just put in the cloud", the opposite of "well it worked for us using a data center" isn't that simple.
      - Spooky23 5y ago
        Say all of those costs are $2,000,000, and you have 25,000 billable endpoints in the datacenter… you’re looking at less than $0.01/hour for that overhead on a unit basis. Obviously, there’s a huge capital investment component too that has to be incorporated. Those costs may be really high if you’re in a growth phase as you need to overbuy capacity. Just to be clear, I’m not arguing that on-prem is magically cheap. :) But it has its place too!
- ctvo 5y ago
  > Not at this rate. And what rate is this? It gets attention because it impacts more people, but AWS / GCP / Azure uptime is still better than what I've seen for small / mid size businesses trying to manage their own infrastructure.
  - iso1631 5y ago
    Multiple outages in a single month.
    - ctvo 5y ago
      Across different regions, availability zones, and services. What's the impact to customers? Today it was a very specific EC2 outage in an AZ in us-east-1. So again, we're talking about cloud providers because of their scope and size, and they're still doing better than MidSizedBank managing their own infrastructure.
dijit 5y ago
What an absolutely pointless comment. Everything fails, we can argue the rate. But I would argue that understanding your constraints is better. if you know that your secret storage system can't survive if a machine goes away: well, you wire redundant paths to the hardware and do memory mirroring and RAID the hell out of the disks. And if it fails you have a standby in place. But if you use AWS Cognito. And it goes down. You're fucked mate.
- f6v 5y ago
  It’s pointless to discuss how crappy cloud is whenever AWS goes down. Most of the businesses relying by the automatic RDS backups or EC2 auto scaling just don’t have time to think about all the underlying tech. I mean, I don’t manually allocate memory for variables anymore either. Do I get screwed when there’s a memory leak? Yes. What do I do about it? Move on.
  - dijit 5y ago
    Then don’t host anything, don’t do software and don’t pretend to be “the future”.
    - Spivak 5y ago
      This makes no sense. This has nothing to do with the tech and more to do with every team's natural push and pull with build over buy. It's completely pointless to respond to someone who didn't get their DoorDash order with "see this is why you should just make food at home." It completely ignores the reason someone chose to order takeout in the first place.
      - noogle 5y ago
        But if someone chose DoorDash over a smaller delivery service because "DoorDash is big and reliable so the x5-10 cost is worth it", and then DoorDash fails repeatedly, it makes sense to ask "is DoorDash really better than using the smaller vendor?"
- 5y ago
RONROC 5y ago
We’re going to be having this same tired, pedantic, round-about conversation when Tesla’s routinely decide to take out a family of four because it mistook a plastic bag for an off-ramp. Commenters will show up like clockwork and say shit like: “What man, it’s not like cars didn’t crash before? Haha” Don’t be dense dude. And definitely don’t pursue a leadership position anytime in the future.
- f6v 5y ago
  Well, anyone non-dense will tell you that most dense thing you can do is say “oh my god run for your lives” whenever there’s an outage. No statics, no cost-benefit analysis. Just commenting “Haha you can’t manage your own raids and ciscos what a noob” makes you a thought leader, yes.
- deanCommie 5y ago
  Tesla fans are annoying, but it is absolutely valid that the safety bar for self-driving cars can't be "100% perfectly safe" - it needs to be "safer than the alternative". The problem with both this example, and the AWS one (it needs to have better availability than your personal home-spun solution, and it does), is that people are amazing at deluding themselves. "Yes, cars are dangerous, because other people can't drive. But I'm a better than average driver" "Yes, other people will build unreliable systems. But I know how to architect for my use case and ensure that for my needs the availability will be higher than AWS's" Both are true* in the micro sense and false in the macro sense. * Not really. 88% of americans think they are "above average" drivers.

Self-managed infrastructure doesn’t fail now?

by f6v 5y ago

Self-managed infrastructure doesn’t fail now?

Not at this rate. I remember we had a power outage in 2006, it actually took one of my services off air. Since then of course that has been rectified, and the loss of a building wouldn't impact on any of the critical, essential or important services I provide.

mbesto 5y ago
> Not at this rate. Source? Has there ever been an industry wide survey that compares availability from "insert average colo/data center operations" with the cloud ones? And I'm not talking about "we have 12 SREs who are based in Cupertino and are all paid top dollar to support a colo"...I'm talking average.
- jandrewrogers 5y ago
  I've worked at a few small companies over the years that had their own significant colos and/or data centers built on the cheap, and only a sysadmin or two to run them. Anecdotally, if the infrastructure is setup right, outages are very rare. Some of these were serving up massive loads at the time. I've done these build-outs a few times in my career and it isn't that hard to do, reliable software is more likely to be an issue. The only significant outage I remember is when the redundant power systems in one DC both failed at the same time for different reasons, which can happen. It is as if the software industry has collectively forgotten how to run basic data center operations. Something that used to be a blue collar skill is now treated like arcane magic.
  - ctvo 5y ago
    > It is as if the software industry has collectively forgotten how to run basic data center operations. Something that used to be a blue collar skill is now treated like arcane magic. It's not arcane magic. It's undifferentiated toil that requires hiring for a different skill set than tech companies generally want to hire for. Of course when you get to a certain size it may make sense to take on this cost. I want us to stop pretending individual actors lack the agency to make their own decisions, and they're all blind to how AWS is charging them a fortune for such simple things they can do themselves. You get value from AWS or you stop using AWS.
- Spooky23 5y ago
  Running a multi-tenant datacenter or hyper scale cloud datacenter is a different business than running a datacenter. The myth of the cost of running facilities on HN is insane - it’s like saying you can’t drive a car unless you hire a formula 1 driver. I worked through the ranks at a large enterprise that ran a “big” datacenter for a decade. The facilities team was about 6 people, average salary around $90k. I can only remember one power interruption affecting more than a rack, caused by a failure during a maintenance event that requires a shutdown for safety reasons. The rest is like any other industrial facility - you have service contracts for the equipment, etc and maintain things. There’s a cost/capability curve that you need to plan around for these matters. You need to make business and engineering decisions based on your actual circumstances. If the answer is automatically “AWS <whatever>“, you’re making a decision to burn dollars for convenience.
  - mbesto 5y ago
    > The facilities team was about 6 people, average salary around $90k. Ok so $540k salaries + benefits, so ~$700k. Then you have transaction costs: - Annual salary increases - Any cost associated with people leaving (severance, hiring, recruiters, HR, HR systems) - Systems that run in the data center (logging, monitoring, etc.) - Procurement costs with changing costs in hardware (silicon shortages, etc.) - Security compliance overhead and associated risks - Finance resources required to capitalize and manage asset allocation - etc. etc. Versus - Click a button and voila it works. - Hire way less engineers to manage the system administrative portion > If the answer is automatically “AWS <whatever>“, you’re making a decision to burn dollars for convenience. 100% AGREE. The answer is always "it depends", but just like people are saying "just put in the cloud", the opposite of "well it worked for us using a data center" isn't that simple.
    - Spooky23 5y ago
      Say all of those costs are $2,000,000, and you have 25,000 billable endpoints in the datacenter… you’re looking at less than $0.01/hour for that overhead on a unit basis. Obviously, there’s a huge capital investment component too that has to be incorporated. Those costs may be really high if you’re in a growth phase as you need to overbuy capacity. Just to be clear, I’m not arguing that on-prem is magically cheap. :) But it has its place too!
ctvo 5y ago
> Not at this rate. And what rate is this? It gets attention because it impacts more people, but AWS / GCP / Azure uptime is still better than what I've seen for small / mid size businesses trying to manage their own infrastructure.
- iso1631 5y ago
  Multiple outages in a single month.
  - ctvo 5y ago
    Across different regions, availability zones, and services. What's the impact to customers? Today it was a very specific EC2 outage in an AZ in us-east-1. So again, we're talking about cloud providers because of their scope and size, and they're still doing better than MidSizedBank managing their own infrastructure.

dijit 5y ago

What an absolutely pointless comment. Everything fails, we can argue the rate. But I would argue that understanding your constraints is better. if you know that your secret storage system can't survive if a machine goes away: well, you wire redundant paths to the hardware and do memory mirroring and RAID the hell out of the disks. And if it fails you have a standby in place. But if you use AWS Cognito. And it goes down. You're fucked mate.

f6v 5y ago
It’s pointless to discuss how crappy cloud is whenever AWS goes down. Most of the businesses relying by the automatic RDS backups or EC2 auto scaling just don’t have time to think about all the underlying tech. I mean, I don’t manually allocate memory for variables anymore either. Do I get screwed when there’s a memory leak? Yes. What do I do about it? Move on.
- dijit 5y ago
  Then don’t host anything, don’t do software and don’t pretend to be “the future”.
  - Spivak 5y ago
    This makes no sense. This has nothing to do with the tech and more to do with every team's natural push and pull with build over buy. It's completely pointless to respond to someone who didn't get their DoorDash order with "see this is why you should just make food at home." It completely ignores the reason someone chose to order takeout in the first place.
    - noogle 5y ago
      But if someone chose DoorDash over a smaller delivery service because "DoorDash is big and reliable so the x5-10 cost is worth it", and then DoorDash fails repeatedly, it makes sense to ask "is DoorDash really better than using the smaller vendor?"
5y ago

RONROC 5y ago

We’re going to be having this same tired, pedantic, round-about conversation when Tesla’s routinely decide to take out a family of four because it mistook a plastic bag for an off-ramp. Commenters will show up like clockwork and say shit like: “What man, it’s not like cars didn’t crash before? Haha” Don’t be dense dude. And definitely don’t pursue a leadership position anytime in the future.

f6v 5y ago
Well, anyone non-dense will tell you that most dense thing you can do is say “oh my god run for your lives” whenever there’s an outage. No statics, no cost-benefit analysis. Just commenting “Haha you can’t manage your own raids and ciscos what a noob” makes you a thought leader, yes.
deanCommie 5y ago
Tesla fans are annoying, but it is absolutely valid that the safety bar for self-driving cars can't be "100% perfectly safe" - it needs to be "safer than the alternative". The problem with both this example, and the AWS one (it needs to have better availability than your personal home-spun solution, and it does), is that people are amazing at deluding themselves. "Yes, cars are dangerous, because other people can't drive. But I'm a better than average driver" "Yes, other people will build unreliable systems. But I know how to architect for my use case and ensure that for my needs the availability will be higher than AWS's" Both are true* in the micro sense and false in the macro sense. * Not really. 88% of americans think they are "above average" drivers.