7 ms·

Think about it this way: 1) Can you make your on prem infrastructure go down less than Amazon's? 2) Is it worth it? In my experience most people grossly unde

Think about it this way: 1) Can you make your on prem infrastructure go down less than Amazon's? 2) Is it worth it? In my experience most people grossly underestimate how expensive it is to create reliable infrastructure and at the same time overestimate how important it is for their services to run uninterrupted. -- EDIT: I am not arguing you shouldn't build your more reliable infrastructure. AWS is just a point on a spectrum of possible compromises between cost and reliability. It might not be right for you. If it is too expensive -- go for cheaper options with less reliability. If it is too unreliable -- go build your own yourself, but make sure you are not making huge mistake because you may not understand what it actually costs to build to AWSs level. For example, personally, not having to focus on infra reliability makes it possible for me to focus on other things that are more important to my company. Do I care about outages? Of course I do, but I understand doing this better than AWS has would cost me huge amount of focus on something that is not core goal of what we are doing. I would rather spend that time thinking how to hire/retain better people and how to make my product better. And adding all that complexity of running this infra to my company would cause entire organisation be less flexible, which is also a cost. So you can't look at cost of running the infra like a bill of materials for parts and services. And if there is an outage it is good to know there is huge organisation there trying to fix it while my small organisation can focus preparing for what to do when it comes back up.

jtc331 5y ago
You’re missing a huge factor: agency.
autosharp 5y ago
Also, you can just take two different amazon regions and hope they don't both go down at the same time. For extra safety, and extra work, you could even take Azure as a backup if you're not locked in with AWS.
- dijit 5y ago
  forgive me repeating myself: AWS Zones are not truly independent of each other. Global services such as route53, Cognito, the default cloud console and Cloudfront are managed out of US-East-1. If us-east-1 is unavailable, as is commonly the case, and you depend on those systems, you are also down. it does not matter if you're in timbuktu-1, you are dead in the water. it is a myth that amazon availability zones are truly independent. please stop blaming the victim, because you can do everything right and still fail if you are not aware of this; and you are perpetuating that unawareness.
  - autosharp 5y ago
    Of course that depends on what services you use and yes, even then there is some remaining correlation just because it is the same host. > are not truly independent of each other Indeed. They are even on the same planet! > please stop blaming the victim Excuse me?
    - dijit 5y ago
      >> are not truly independent of each other > Indeed. They are even on the same planet! Clever bastard, aren't you. >> please stop blaming the victim > Excuse me? "If you're affected by us-east-1 outages then you're not hosting in other regions and you're doing it wrong". Except: You can be affected by this outage if you did everything right. You're putting blame on people being down for not being hosted in different regions when it would not help them. You've effectively shifted blame away from Amazon and onto the person who cannot control their uptime by doing what you said.
      - autosharp 5y ago
        > "If you're affected by us-east-1 outages then you're not hosting in other regions and you're doing it wrong". You are attributing a quote to me which I never expressed, nor was that expressed elsewhere in this thread. You are even using quotation marks.... I certainly didn't mean to blame anyone. You appear to see this AWS issue as one of victims and victimizers. I was just trying to point out an agency that people may have in some situations.
        dijit 5y ago
        I was not quoting you, I was echoing your sentiment back to you with different words to see if you disagreed with it. I was just re-wording the sentiment. Let me quote you properly. > Also, you can just take two different amazon regions and hope they don't both go down at the same time. Do you see how replacing that in my comments does not change the sentiment?
Nextgrid 5y ago
> Can you make your on prem infrastructure go down less than Amazon's? Obviously depends on what you need, but for a small to medium web app that needs a load-balancer, a few app servers, a database and a cache, yes absolutely - all of these have been solved problems for over a decade and aren't rocket science to install & maintain. > Is it worth it? I'd argue that the "worth" would be less about immunity to occasional outages but the continuous savings when it comes to price per performance & not having to pay for bandwidth. > overestimate how important it is for their services to run uninterrupted. Agreed. However when running on-prem, should your service go down and you need it back up, you can do something about it. With the cloud, you have no choice but to wait.
- badams2527 5y ago
  Human capital side would disagree with that I think. You're assuming the organization which owns this small/medium web app has the personnel already on staff to handle such a thing. If you're outsourcing that, you'd likely have to pay a boatload just for someone to be available for help, let alone the actual tasks themselves. Like you said, if you're on-prem and something goes down, you can do something. But you've gotta have the personnel to actually do something. That said, I think you're spot-on as long as you have the skillset already.
  - Nextgrid 5y ago
    > Human capital side would disagree with that I think I hear this argument a lot, but every startup I've been involved with had a full-time DevOps engineer wrangling Terraform & YAML files - that same engineer can be assigned to manage the bare-metal infrastructure.
    - dragonwriter 5y ago
      > I hear this argument a lot, but every startup I've been involved with had a full-time DevOps engineer wrangling Terraform & YAML files - that same engineer can be assigned to manage the bare-metal infrastructure. Bare metal infrastructure requires a lot more management at any given scale. I mean, you can run stuff that lets you do part of the management the same as cloud resources, but you also have to then manage that software and manage the hardware.
jerf 5y ago
I think if you put a bit of effort into classifying importance, you can likely justify backing up certain critical systems in more than one way. Let "the cloud" handle everyone's desktop backups and all the ancillary systems you don't really need immediately to do business, but certain important systems should perhaps be backed up both to the cloud and locally, like Windows Domain Controllers and other things you can't do anything without. Backup is cheap when you're focused about what you're backing up. In this case, the game isn't "going down less than Amazon", it's about going down uncorrelated to Amazon. Though that's getting harder! "In more than one way" doesn't have to be local, but it may be across multiple cloud services. Still, "local" is nice in that it doesn't require the Internet. ("The Internet" doesn't tend to go down, but the portion you are on certainly can.) Of course, as workers disperse, "local" means less and less nowadays.
- kaashif 5y ago
  > In this case, the game isn't "going down less than Amazon", it's about going down uncorrelated to Amazon. It's possible to go down in a mostly uncorrelated way to Amazon by just being down all the time. Obviously this is implicit in your comment, but I'll say it anyway: your backups need to actually work when you need them. You need to test them (really test them) to make sure they're not secretly non-functional in some subtle way when Amazon is really down.
woodruffw 5y ago
Not my company, but I work with another company that does (nearly?) all of their infrastructure on premise. They have pretty great uptime, in a large part because they're not dependent on the 3-4 global state mechanisms that consistently cause outages with cloud providers (DNS, BGP, AWS's role management/control plane, &c.). I think you're right about what we over- & under-estimate, but that we also under-estimate the inflection point for when it makes sense to begin relying on major cloud services. Put another way: we over-estimate our requirements, causing us to pessimistically reach for services that have problems that we'd otherwise never have.
dgudkov 5y ago
1) Can you make your on prem infrastructure go down less than Amazon's? It's now hard to say how frequently Amazon's infrastructure goes down. The incident rate seems to have accelerated.
patentatt 5y ago
On the other hand, perhaps the large cloud providers bring a level of complexity that outweighs their skills at keeping everything up. What I mean is, a basic redundancy and failover setup with two data centers is kind of straightforward. Sure you need a person on call 24/7 to oversee it, but it's conceptually not that complicated. And if you're running bare metal, you get a surprising level of performance per dollar and rack unit. On the other hand, the big clouds are immensely complex with multiple layers of software defined networking, millions of tenants, thousands of employees, acres of floor space, org charts, etc. If you're running your own infra as one competent sysadmin, you know nobody else in another department will push a networking code change that will break your shit in the middle of the night. Maybe it's not right for everyone, but it's not unreasonable to go on prem in 2021 despite the popular opinions otherwise. Source: my company runs on prem and routinely has 100% uptime years. Most unplanned downtime occurs early on a Sunday morning following a planned action during a maintenance window.
- sgarland 5y ago
  I was and continue to be surprised how reliable even old servers are. I run a small homelab (Debian VMs on Proxmox; a Docker host, a jumpbox, a NAS running ZFS, etc.) on seven year old hardware, and all of my problems are self-imposed. If I leave everything alone, it just works. As a counterpoint, though, my last place had a large Java app, split between colo'd metal and AWS. Seemed like the colo'd stuff failed more (bad RAM mostly, a few CPUs, and an occasional PSU). Entirely anecdotal.
Retric 5y ago
It really spends on how reliable you need to be. Don’t forget you get downtime from both AWS and your own issues so even 4 9’s is off the table with pure AWS. If you need to be more reliable than AWS you need to run a hybrid inside and outside of AWS which means most of the advantages of running on AWS goes away.
- nostrebored 5y ago
  Very untrue. Many businesses with 4 9 SLAs are all in on AWS. It requires active/active setups though!
  - Retric 5y ago
    Many business claim 4 9 SLAs on AWS, but that doesn’t mean they actually provide it. It’s simply a question of what the penalties of failing to reach their SLA is.
ocdtrekkie 5y ago
My on prem infrastructure goes down drastically less than Amazon's. ...My home Internet even is scoring better than Amazon right now, in fact. Yours probably is too.
- lmilcin 5y ago
  I have a bolt lying on my desk. It hasn't failed since 1970 when it was produced. It must have been built better than Space Shuttle, then.
  - dijit 5y ago
    Sysadmin 102: Simple Systems Fail Less Often.
StreamBright 5y ago
3) Could you hire talent that can build the thing? In my experience problem number 3 is the hardest to solve.
pkulak 5y ago
> Can you make your on prem infrastructure go down less than Amazon's? Over the last two years, my track record has destroyed AWS. I've got a single Mac Mini with two VMs on it, plugged in to a UPS with enough power to keep it running for about three hours. It's never had a second of unplanned downtime. About 15 years ago I got sick of maintaining my own stuff. I stopped building Linux desktops and bought an Apple laptop. I moved my email, calendars, contacts, chat, photos, etc, to Google. But lately I've swung 180 degrees and have been undoing all those decisions. It's not as much of a PITA as I remember. Maybe I'm better at it now? Or maybe it will become a PITA and I'll swing right back. EDIT: I realize you're talking in a commercial sense and I'm talking about a homelab sense. Still, take my anecdote for what it's worth. :D

jtc331 5y ago

You’re missing a huge factor: agency.

autosharp 5y ago

Also, you can just take two different amazon regions and hope they don't both go down at the same time. For extra safety, and extra work, you could even take Azure as a backup if you're not locked in with AWS.

dijit 5y ago
forgive me repeating myself: AWS Zones are not truly independent of each other. Global services such as route53, Cognito, the default cloud console and Cloudfront are managed out of US-East-1. If us-east-1 is unavailable, as is commonly the case, and you depend on those systems, you are also down. it does not matter if you're in timbuktu-1, you are dead in the water. it is a myth that amazon availability zones are truly independent. please stop blaming the victim, because you can do everything right and still fail if you are not aware of this; and you are perpetuating that unawareness.
- autosharp 5y ago
  Of course that depends on what services you use and yes, even then there is some remaining correlation just because it is the same host. > are not truly independent of each other Indeed. They are even on the same planet! > please stop blaming the victim Excuse me?
  - dijit 5y ago
    >> are not truly independent of each other > Indeed. They are even on the same planet! Clever bastard, aren't you. >> please stop blaming the victim > Excuse me? "If you're affected by us-east-1 outages then you're not hosting in other regions and you're doing it wrong". Except: You can be affected by this outage if you did everything right. You're putting blame on people being down for not being hosted in different regions when it would not help them. You've effectively shifted blame away from Amazon and onto the person who cannot control their uptime by doing what you said.
    - autosharp 5y ago
      > "If you're affected by us-east-1 outages then you're not hosting in other regions and you're doing it wrong". You are attributing a quote to me which I never expressed, nor was that expressed elsewhere in this thread. You are even using quotation marks.... I certainly didn't mean to blame anyone. You appear to see this AWS issue as one of victims and victimizers. I was just trying to point out an agency that people may have in some situations.
      - dijit 5y ago
        I was not quoting you, I was echoing your sentiment back to you with different words to see if you disagreed with it. I was just re-wording the sentiment. Let me quote you properly. > Also, you can just take two different amazon regions and hope they don't both go down at the same time. Do you see how replacing that in my comments does not change the sentiment?

Nextgrid 5y ago

> Can you make your on prem infrastructure go down less than Amazon's? Obviously depends on what you need, but for a small to medium web app that needs a load-balancer, a few app servers, a database and a cache, yes absolutely - all of these have been solved problems for over a decade and aren't rocket science to install & maintain. > Is it worth it? I'd argue that the "worth" would be less about immunity to occasional outages but the continuous savings when it comes to price per performance & not having to pay for bandwidth. > overestimate how important it is for their services to run uninterrupted. Agreed. However when running on-prem, should your service go down and you need it back up, you can do something about it. With the cloud, you have no choice but to wait.

badams2527 5y ago
Human capital side would disagree with that I think. You're assuming the organization which owns this small/medium web app has the personnel already on staff to handle such a thing. If you're outsourcing that, you'd likely have to pay a boatload just for someone to be available for help, let alone the actual tasks themselves. Like you said, if you're on-prem and something goes down, you can do something. But you've gotta have the personnel to actually do something. That said, I think you're spot-on as long as you have the skillset already.
- Nextgrid 5y ago
  > Human capital side would disagree with that I think I hear this argument a lot, but every startup I've been involved with had a full-time DevOps engineer wrangling Terraform & YAML files - that same engineer can be assigned to manage the bare-metal infrastructure.
  - dragonwriter 5y ago
    > I hear this argument a lot, but every startup I've been involved with had a full-time DevOps engineer wrangling Terraform & YAML files - that same engineer can be assigned to manage the bare-metal infrastructure. Bare metal infrastructure requires a lot more management at any given scale. I mean, you can run stuff that lets you do part of the management the same as cloud resources, but you also have to then manage that software and manage the hardware.

jerf 5y ago

I think if you put a bit of effort into classifying importance, you can likely justify backing up certain critical systems in more than one way. Let "the cloud" handle everyone's desktop backups and all the ancillary systems you don't really need immediately to do business, but certain important systems should perhaps be backed up both to the cloud and locally, like Windows Domain Controllers and other things you can't do anything without. Backup is cheap when you're focused about what you're backing up. In this case, the game isn't "going down less than Amazon", it's about going down uncorrelated to Amazon. Though that's getting harder! "In more than one way" doesn't have to be local, but it may be across multiple cloud services. Still, "local" is nice in that it doesn't require the Internet. ("The Internet" doesn't tend to go down, but the portion you are on certainly can.) Of course, as workers disperse, "local" means less and less nowadays.

kaashif 5y ago
> In this case, the game isn't "going down less than Amazon", it's about going down uncorrelated to Amazon. It's possible to go down in a mostly uncorrelated way to Amazon by just being down all the time. Obviously this is implicit in your comment, but I'll say it anyway: your backups need to actually work when you need them. You need to test them (really test them) to make sure they're not secretly non-functional in some subtle way when Amazon is really down.

woodruffw 5y ago

Not my company, but I work with another company that does (nearly?) all of their infrastructure on premise. They have pretty great uptime, in a large part because they're not dependent on the 3-4 global state mechanisms that consistently cause outages with cloud providers (DNS, BGP, AWS's role management/control plane, &c.). I think you're right about what we over- & under-estimate, but that we also under-estimate the inflection point for when it makes sense to begin relying on major cloud services. Put another way: we over-estimate our requirements, causing us to pessimistically reach for services that have problems that we'd otherwise never have.

dgudkov 5y ago

1) Can you make your on prem infrastructure go down less than Amazon's? It's now hard to say how frequently Amazon's infrastructure goes down. The incident rate seems to have accelerated.

patentatt 5y ago

On the other hand, perhaps the large cloud providers bring a level of complexity that outweighs their skills at keeping everything up. What I mean is, a basic redundancy and failover setup with two data centers is kind of straightforward. Sure you need a person on call 24/7 to oversee it, but it's conceptually not that complicated. And if you're running bare metal, you get a surprising level of performance per dollar and rack unit. On the other hand, the big clouds are immensely complex with multiple layers of software defined networking, millions of tenants, thousands of employees, acres of floor space, org charts, etc. If you're running your own infra as one competent sysadmin, you know nobody else in another department will push a networking code change that will break your shit in the middle of the night. Maybe it's not right for everyone, but it's not unreasonable to go on prem in 2021 despite the popular opinions otherwise. Source: my company runs on prem and routinely has 100% uptime years. Most unplanned downtime occurs early on a Sunday morning following a planned action during a maintenance window.

sgarland 5y ago
I was and continue to be surprised how reliable even old servers are. I run a small homelab (Debian VMs on Proxmox; a Docker host, a jumpbox, a NAS running ZFS, etc.) on seven year old hardware, and all of my problems are self-imposed. If I leave everything alone, it just works. As a counterpoint, though, my last place had a large Java app, split between colo'd metal and AWS. Seemed like the colo'd stuff failed more (bad RAM mostly, a few CPUs, and an occasional PSU). Entirely anecdotal.

Retric 5y ago

It really spends on how reliable you need to be. Don’t forget you get downtime from both AWS and your own issues so even 4 9’s is off the table with pure AWS. If you need to be more reliable than AWS you need to run a hybrid inside and outside of AWS which means most of the advantages of running on AWS goes away.

nostrebored 5y ago
Very untrue. Many businesses with 4 9 SLAs are all in on AWS. It requires active/active setups though!
- Retric 5y ago
  Many business claim 4 9 SLAs on AWS, but that doesn’t mean they actually provide it. It’s simply a question of what the penalties of failing to reach their SLA is.

ocdtrekkie 5y ago

My on prem infrastructure goes down drastically less than Amazon's. ...My home Internet even is scoring better than Amazon right now, in fact. Yours probably is too.

lmilcin 5y ago
I have a bolt lying on my desk. It hasn't failed since 1970 when it was produced. It must have been built better than Space Shuttle, then.
- dijit 5y ago
  Sysadmin 102: Simple Systems Fail Less Often.

StreamBright 5y ago

3) Could you hire talent that can build the thing? In my experience problem number 3 is the hardest to solve.

pkulak 5y ago

> Can you make your on prem infrastructure go down less than Amazon's? Over the last two years, my track record has destroyed AWS. I've got a single Mac Mini with two VMs on it, plugged in to a UPS with enough power to keep it running for about three hours. It's never had a second of unplanned downtime. About 15 years ago I got sick of maintaining my own stuff. I stopped building Linux desktops and bought an Apple laptop. I moved my email, calendars, contacts, chat, photos, etc, to Google. But lately I've swung 180 degrees and have been undoing all those decisions. It's not as much of a PITA as I remember. Maybe I'm better at it now? Or maybe it will become a PITA and I'll swing right back. EDIT: I realize you're talking in a commercial sense and I'm talking about a homelab sense. Still, take my anecdote for what it's worth. :D