12 ms·

Me: Hesitation at last job moving absolutely everything (including backups) to AWS because if it goes down it's a problem I'm a firm believer in some kind of p

Me: Hesitation at last job moving absolutely everything (including backups) to AWS because if it goes down it's a problem I'm a firm believer in some kind of physical/easily accessible backup. Coworkers: "You're an f'n idiot. Amazon and Facebook don't go down, you're holding us back!" <-Quite literally their words. Me: leaves cause that treatment was the final straw Amazon and Facebook both go down within a month of each other, and supposedly they needed backups Them: shocked pikachu face

dookahku 5y ago
Send Your former colleagues a group email asking how it is
xwdv 5y ago
You’re still in the wrong, don’t be so smug. These few downtimes are no big deal in the grand scheme of things, and your proposed solution would have been more work and headaches for little to no realizable gains, and not to mention the cybersecurity ramifications. Quite frankly, they are probably glad that you’re gone and not around to gloat about every trivial bit of downtime.
- locallost 5y ago
  They're not gloating and also not smug. There's not even a 'hehe' in the post.
lmilcin 5y ago
Think about it this way: 1) Can you make your on prem infrastructure go down less than Amazon's? 2) Is it worth it? In my experience most people grossly underestimate how expensive it is to create reliable infrastructure and at the same time overestimate how important it is for their services to run uninterrupted. -- EDIT: I am not arguing you shouldn't build your more reliable infrastructure. AWS is just a point on a spectrum of possible compromises between cost and reliability. It might not be right for you. If it is too expensive -- go for cheaper options with less reliability. If it is too unreliable -- go build your own yourself, but make sure you are not making huge mistake because you may not understand what it actually costs to build to AWSs level. For example, personally, not having to focus on infra reliability makes it possible for me to focus on other things that are more important to my company. Do I care about outages? Of course I do, but I understand doing this better than AWS has would cost me huge amount of focus on something that is not core goal of what we are doing. I would rather spend that time thinking how to hire/retain better people and how to make my product better. And adding all that complexity of running this infra to my company would cause entire organisation be less flexible, which is also a cost. So you can't look at cost of running the infra like a bill of materials for parts and services. And if there is an outage it is good to know there is huge organisation there trying to fix it while my small organisation can focus preparing for what to do when it comes back up.
- jtc331 5y ago
  You’re missing a huge factor: agency.
- autosharp 5y ago
  Also, you can just take two different amazon regions and hope they don't both go down at the same time. For extra safety, and extra work, you could even take Azure as a backup if you're not locked in with AWS.
  - dijit 5y ago
    forgive me repeating myself: AWS Zones are not truly independent of each other. Global services such as route53, Cognito, the default cloud console and Cloudfront are managed out of US-East-1. If us-east-1 is unavailable, as is commonly the case, and you depend on those systems, you are also down. it does not matter if you're in timbuktu-1, you are dead in the water. it is a myth that amazon availability zones are truly independent. please stop blaming the victim, because you can do everything right and still fail if you are not aware of this; and you are perpetuating that unawareness.
    - autosharp 5y ago
      Of course that depends on what services you use and yes, even then there is some remaining correlation just because it is the same host. > are not truly independent of each other Indeed. They are even on the same planet! > please stop blaming the victim Excuse me?
      - dijit 5y ago
        >> are not truly independent of each other > Indeed. They are even on the same planet! Clever bastard, aren't you. >> please stop blaming the victim > Excuse me? "If you're affected by us-east-1 outages then you're not hosting in other regions and you're doing it wrong". Except: You can be affected by this outage if you did everything right. You're putting blame on people being down for not being hosted in different regions when it would not help them. You've effectively shifted blame away from Amazon and onto the person who cannot control their uptime by doing what you said.
kalleth 5y ago
I'd be surprised if they needed backups for a few hours of downtime with (reportedly) complete recovery where no data was corrupted. There are industries where this would be required, and it's possible I guess, but neither of these downtime events were "data loss" events, just availability events for short-ish periods of time that wouldn't - for me - result in activating our DR plans. I must admit that I do always try and maintain a separate data backup for true disaster recovery scenarios - but those are mainly focused around AWS locking me out of our AWS account (and hence we can't access our data or backups) or recovering from a crypto scam hack that also corrupts on-platform backups, for example.
- aeonflux 5y ago
  I once had to argue that we still do need backup even though S3 has redundancy. They laughed when I mentioned a possible lock-up from AWS (even due to a mistake or whatever). I asked what if we delete data from app by mistake? They told me we need to be careful not to do that. I guess I am getting more and more tired of arrogant 25 years old programmers with 1-2 years in industry and no experience.
  - manquer 5y ago
    S3 and (others) have version history that can be enabled. If you have to take care of availablity and redundancy and delete protection and backups then why pay the premium S3 is charging ? Either you don't trust the cloud and you can run NAS or equivalent (with s3 APIs easily today) much cheaper or trust them to keep your data safe and available. No point in investing in S3 and then doing it again yourself.
    - ncallaway 5y ago
      > No point in investing in S3 and then doing it again yourself. I mean that's just obviously wrong, though. There is a point. > Either you don't trust the cloud and you can run NAS or equivalent (with s3 APIs easily today) much cheaper or trust them to keep your data safe and available. What if you trust the cloud 90%, and you trust yourself 90%, and you think it's likely that the failure cases between the two are likely to be independent? Then it seems like the smart decision would be to do both. Your position is basically arguing that redundant systems are never necessary, because "either you trust A or you trust B, why do both?" If it's absolutely critical that you don't suffer a particular failure, then having redundant systems is very wise.
rafale 5y ago
Did u file a complaint on the use of swear words?
jmartrican 5y ago
Seems like multi-cloud solution might be the way to go.
- nier 5y ago
  All while making sure that these cloud solutions are not inter-dependent and that there are redundant paths to access these services.
- thedougd 5y ago
  I doubt it. The complexity of multi-cloud will also give you downtime. Most of the folks impacted by cloud outages do not have highly available systems in place. Perhaps, for their business, the cost doesn't justify the outcome. If you need high uptime for instances, build your system to be highly available and leverage the fault domain constructs your provider offers (placement groups, availability zones, regions, load balancing, DNS routing, autoscaling groups, service discovery, etc). For instances, double down and use spot instance and maximum lifetimes in your groups so that you're continuously validating your application can recovery from instance interruptions. If you're heavy on applications that leverage cloud APIs, such as is often the case with labmdas, then strongly consider multi-region active/active as API outages tend to cross AZ's and impact the entire region.
  - jmartrican 5y ago
    Agreed it is hard for those reason you specified. To do it, first I would not use any cloud features that cannot be easily setup in another cloud. So no lambdas. Just k8s clusters, maybe DBs if they can be setup to backup between clouds. I was able to migrate from AWS k8s to DO K8S very easily.... just pointed my k8s configs to the new cluster (plus configuring the DO load balancers). In my case, I need the dynamic DNS (havnt looked into it yet), auto-scaling is already setup with k8s, and the DB backups between DBs (next project).
uvdn7 5y ago
You could have just showed them historical data of both companies being unavailable for extended amount of time. What happened in the past few months is not new.
- joana035 5y ago
  "just", as if you never had to argument against aws fanboys...
  - meshaneian 5y ago
    As a pragmatic AWS fan, +1 this. Disposable distributed hybrid multi-cloud architecture FTW.
mattl 5y ago
Backup to rsync.net
davewritescode 5y ago
You’re not wrong but there’s ways to do backups properly in AWS and I’m not aware of there ever being an incident where AWS has lost data. It’s not a bad idea store backups offline but costs might make that an expensive proposition.
- numbsafari 5y ago
  S3 isn't perfect. Read the fine print. I've had buckets and objects disappear into the ether. It is exceedingly rare, but it's not impossible. Offline/alt-cloud backups are probably a lot cheaper than you think, and will win you points during any audit.
  - thraxil 5y ago
    > Offline/alt-cloud backups are probably a lot cheaper than you think, and will win you points during any audit. With the caveat that you're going to have to implement all your access controls, monitoring and compliance mechanisms on those alternate backups. No point winning points during an audit for having backups outside AWS if you lose even more points for "backups weren't properly secured against unauthorized access". And you're regularly restoring from those alternate backups as well to check their integrity, right?
    - numbsafari 5y ago
      Well, obviously, it goes without saying. But none of that changes the fact that you shouldn't put all your eggs in one basket.
numbsafari 5y ago
Today's gentle reminder that there are things other than network or service outages that can and do occur that might necessitate an outside backup. What happens if AWS or [insert other megacloud] decides your account needs to be nuked from orbit due to a hack or some other confusion? We almost had this happen over the summer because of a problem with our bank's ability to process ACH payments. Very frustrating experience. Still isn't fully resolved. What happens if an admin account is taken over and your account gets screwed up? What happens if an admin loses his shit and blows up your account? What happens if your software has a bug that destroys a bunch of your data or fubars your account? There's a ton of cases where having at least a simple replica of your S3 buckets into a third-party cloud could prove highly valuable.
- hinkley 5y ago
  I would make a friendly wager that AWS user IDs don't contain check digits, let alone bullet proof ones (simple check digits don't guard against transposition errors). And that somewhere, someone can manually enter an account to delete, and that one of us will eventually have an account numbered XXX1234 and some idiot with account XXX1243 will legitimately earn an account deletion, but we'll be the ones who wake up to bad news.
- btown 5y ago
  Would you be able to expand at all about the ACH/AWS connection, obviously without identifying details? Was it just a miscommunication around AWS billing and them thinking you weren't paying? Or did AWS somehow put itself in the middle of, or react to, your use of ACH payment processing for *non-AWS* receivables or payables? If the latter, that's a business risk I'd never even thought about. I'm not even sure how they'd know. But I'm thoughtful that things like the MATCH list [0] exist, and how easily a merchant can accidentally wind up on these lists from either human error or a small amount of high-value chargebacks. If cloud providers are somehow paying attention to merchant services reputation, that would be very scary for many businesses! [0] https://www.merchantmaverick.com/learning-terminated-merchant-file-tmf-aka-match-list/ https://www.merchantmaverick.com/learning-terminated-merchan...
  - 5y ago
fatnoah 5y ago
My last startup migrated from Verizon Terremark after the healthcare.gov fiasco several years ago. We also suffered from that massive outage and that was the final straw in migrating to AWS. At AWS, we built a few layers of redundant infrastructure with mulit-AZ availability within a region and then global availability across multiple regions. All this was done at roughly half the cost of the traditional hosting, even when including the additional person-hours required to maintain it on our end. Keeping our infra simple helped that work, and it's literally been years since an outage caused by any AWS issues, even though there have been several large AWS events.
- zymhan 5y ago
  Indeed, if you only deploy resources in us-east1, or any other single region, you're risking the occasional downtime. I'd wager that will still give you more uptime than a physically-hosted solution for the same cost.
  - hinkley 5y ago
    Honestly, I have an app in production that isn't completely hardened against single zone outages. There was pressure to turn off some redundancy in our caching infra, and not every backend service we call is free of tenant affinity, so we could well lose at least 1/3rd of our customers in a single AZ failure in the wrong region, or have huge latency issues for all of our tenants based on high cache miss rates. Having written this, I'm going to ping our SME on the cache replication and remind him that since the last time he benchmarked it, we've upgraded to a newer generation of EC2 instances that has lower latency, and could he please run those numbers again.
  - fatnoah 5y ago
    As I discovered many years ago when our infra was only in US-EAST1, failures were also easier to explain since many, many other companies would be offline as well. It made it more of an "Internet problem" than our own company's problem. For whatever reason, customers were far more likely to accept those kinds of outages.
- hinkley 5y ago
kburman 5y ago
AWS or Google or any other reputable cloud provider are still far more better options then your local backup. Only way I see you losing your data is account getting locked.
hinkley 5y ago
Have you contacted them to see how things are going? Maybe a cheery note asking how the team is doing, sent right in the middle of an outage. Passive aggressive? As hell. Cathartic? Damn skippy.
whydoyoucare 5y ago
It reminds me of the old adage: "Two is one, one is none. Have a backup. Always."
deleted 5y ago
[deleted]

dookahku 5y ago

Send Your former colleagues a group email asking how it is

xwdv 5y ago

You’re still in the wrong, don’t be so smug. These few downtimes are no big deal in the grand scheme of things, and your proposed solution would have been more work and headaches for little to no realizable gains, and not to mention the cybersecurity ramifications. Quite frankly, they are probably glad that you’re gone and not around to gloat about every trivial bit of downtime.

locallost 5y ago
They're not gloating and also not smug. There's not even a 'hehe' in the post.

lmilcin 5y ago

Think about it this way: 1) Can you make your on prem infrastructure go down less than Amazon's? 2) Is it worth it? In my experience most people grossly underestimate how expensive it is to create reliable infrastructure and at the same time overestimate how important it is for their services to run uninterrupted. -- EDIT: I am not arguing you shouldn't build your more reliable infrastructure. AWS is just a point on a spectrum of possible compromises between cost and reliability. It might not be right for you. If it is too expensive -- go for cheaper options with less reliability. If it is too unreliable -- go build your own yourself, but make sure you are not making huge mistake because you may not understand what it actually costs to build to AWSs level. For example, personally, not having to focus on infra reliability makes it possible for me to focus on other things that are more important to my company. Do I care about outages? Of course I do, but I understand doing this better than AWS has would cost me huge amount of focus on something that is not core goal of what we are doing. I would rather spend that time thinking how to hire/retain better people and how to make my product better. And adding all that complexity of running this infra to my company would cause entire organisation be less flexible, which is also a cost. So you can't look at cost of running the infra like a bill of materials for parts and services. And if there is an outage it is good to know there is huge organisation there trying to fix it while my small organisation can focus preparing for what to do when it comes back up.

jtc331 5y ago
You’re missing a huge factor: agency.
autosharp 5y ago
Also, you can just take two different amazon regions and hope they don't both go down at the same time. For extra safety, and extra work, you could even take Azure as a backup if you're not locked in with AWS.
- dijit 5y ago
  forgive me repeating myself: AWS Zones are not truly independent of each other. Global services such as route53, Cognito, the default cloud console and Cloudfront are managed out of US-East-1. If us-east-1 is unavailable, as is commonly the case, and you depend on those systems, you are also down. it does not matter if you're in timbuktu-1, you are dead in the water. it is a myth that amazon availability zones are truly independent. please stop blaming the victim, because you can do everything right and still fail if you are not aware of this; and you are perpetuating that unawareness.
  - autosharp 5y ago
    Of course that depends on what services you use and yes, even then there is some remaining correlation just because it is the same host. > are not truly independent of each other Indeed. They are even on the same planet! > please stop blaming the victim Excuse me?
    - dijit 5y ago
      >> are not truly independent of each other > Indeed. They are even on the same planet! Clever bastard, aren't you. >> please stop blaming the victim > Excuse me? "If you're affected by us-east-1 outages then you're not hosting in other regions and you're doing it wrong". Except: You can be affected by this outage if you did everything right. You're putting blame on people being down for not being hosted in different regions when it would not help them. You've effectively shifted blame away from Amazon and onto the person who cannot control their uptime by doing what you said.

kalleth 5y ago

I'd be surprised if they needed backups for a few hours of downtime with (reportedly) complete recovery where no data was corrupted. There are industries where this would be required, and it's possible I guess, but neither of these downtime events were "data loss" events, just availability events for short-ish periods of time that wouldn't - for me - result in activating our DR plans. I must admit that I do always try and maintain a separate data backup for true disaster recovery scenarios - but those are mainly focused around AWS locking me out of our AWS account (and hence we can't access our data or backups) or recovering from a crypto scam hack that also corrupts on-platform backups, for example.

aeonflux 5y ago
I once had to argue that we still do need backup even though S3 has redundancy. They laughed when I mentioned a possible lock-up from AWS (even due to a mistake or whatever). I asked what if we delete data from app by mistake? They told me we need to be careful not to do that. I guess I am getting more and more tired of arrogant 25 years old programmers with 1-2 years in industry and no experience.
- manquer 5y ago
  S3 and (others) have version history that can be enabled. If you have to take care of availablity and redundancy and delete protection and backups then why pay the premium S3 is charging ? Either you don't trust the cloud and you can run NAS or equivalent (with s3 APIs easily today) much cheaper or trust them to keep your data safe and available. No point in investing in S3 and then doing it again yourself.
  - ncallaway 5y ago
    > No point in investing in S3 and then doing it again yourself. I mean that's just obviously wrong, though. There is a point. > Either you don't trust the cloud and you can run NAS or equivalent (with s3 APIs easily today) much cheaper or trust them to keep your data safe and available. What if you trust the cloud 90%, and you trust yourself 90%, and you think it's likely that the failure cases between the two are likely to be independent? Then it seems like the smart decision would be to do both. Your position is basically arguing that redundant systems are never necessary, because "either you trust A or you trust B, why do both?" If it's absolutely critical that you don't suffer a particular failure, then having redundant systems is very wise.

rafale 5y ago

Did u file a complaint on the use of swear words?

jmartrican 5y ago

Seems like multi-cloud solution might be the way to go.

nier 5y ago
All while making sure that these cloud solutions are not inter-dependent and that there are redundant paths to access these services.
thedougd 5y ago
I doubt it. The complexity of multi-cloud will also give you downtime. Most of the folks impacted by cloud outages do not have highly available systems in place. Perhaps, for their business, the cost doesn't justify the outcome. If you need high uptime for instances, build your system to be highly available and leverage the fault domain constructs your provider offers (placement groups, availability zones, regions, load balancing, DNS routing, autoscaling groups, service discovery, etc). For instances, double down and use spot instance and maximum lifetimes in your groups so that you're continuously validating your application can recovery from instance interruptions. If you're heavy on applications that leverage cloud APIs, such as is often the case with labmdas, then strongly consider multi-region active/active as API outages tend to cross AZ's and impact the entire region.
- jmartrican 5y ago
  Agreed it is hard for those reason you specified. To do it, first I would not use any cloud features that cannot be easily setup in another cloud. So no lambdas. Just k8s clusters, maybe DBs if they can be setup to backup between clouds. I was able to migrate from AWS k8s to DO K8S very easily.... just pointed my k8s configs to the new cluster (plus configuring the DO load balancers). In my case, I need the dynamic DNS (havnt looked into it yet), auto-scaling is already setup with k8s, and the DB backups between DBs (next project).

uvdn7 5y ago

You could have just showed them historical data of both companies being unavailable for extended amount of time. What happened in the past few months is not new.

joana035 5y ago
"just", as if you never had to argument against aws fanboys...
- meshaneian 5y ago
  As a pragmatic AWS fan, +1 this. Disposable distributed hybrid multi-cloud architecture FTW.

mattl 5y ago

Backup to rsync.net

davewritescode 5y ago

You’re not wrong but there’s ways to do backups properly in AWS and I’m not aware of there ever being an incident where AWS has lost data. It’s not a bad idea store backups offline but costs might make that an expensive proposition.

numbsafari 5y ago
S3 isn't perfect. Read the fine print. I've had buckets and objects disappear into the ether. It is exceedingly rare, but it's not impossible. Offline/alt-cloud backups are probably a lot cheaper than you think, and will win you points during any audit.
- thraxil 5y ago
  > Offline/alt-cloud backups are probably a lot cheaper than you think, and will win you points during any audit. With the caveat that you're going to have to implement all your access controls, monitoring and compliance mechanisms on those alternate backups. No point winning points during an audit for having backups outside AWS if you lose even more points for "backups weren't properly secured against unauthorized access". And you're regularly restoring from those alternate backups as well to check their integrity, right?
  - numbsafari 5y ago
    Well, obviously, it goes without saying. But none of that changes the fact that you shouldn't put all your eggs in one basket.

numbsafari 5y ago

Today's gentle reminder that there are things other than network or service outages that can and do occur that might necessitate an outside backup. What happens if AWS or [insert other megacloud] decides your account needs to be nuked from orbit due to a hack or some other confusion? We almost had this happen over the summer because of a problem with our bank's ability to process ACH payments. Very frustrating experience. Still isn't fully resolved. What happens if an admin account is taken over and your account gets screwed up? What happens if an admin loses his shit and blows up your account? What happens if your software has a bug that destroys a bunch of your data or fubars your account? There's a ton of cases where having at least a simple replica of your S3 buckets into a third-party cloud could prove highly valuable.

hinkley 5y ago
I would make a friendly wager that AWS user IDs don't contain check digits, let alone bullet proof ones (simple check digits don't guard against transposition errors). And that somewhere, someone can manually enter an account to delete, and that one of us will eventually have an account numbered XXX1234 and some idiot with account XXX1243 will legitimately earn an account deletion, but we'll be the ones who wake up to bad news.
btown 5y ago
Would you be able to expand at all about the ACH/AWS connection, obviously without identifying details? Was it just a miscommunication around AWS billing and them thinking you weren't paying? Or did AWS somehow put itself in the middle of, or react to, your use of ACH payment processing for *non-AWS* receivables or payables? If the latter, that's a business risk I'd never even thought about. I'm not even sure how they'd know. But I'm thoughtful that things like the MATCH list [0] exist, and how easily a merchant can accidentally wind up on these lists from either human error or a small amount of high-value chargebacks. If cloud providers are somehow paying attention to merchant services reputation, that would be very scary for many businesses! [0] https://www.merchantmaverick.com/learning-terminated-merchant-file-tmf-aka-match-list/ https://www.merchantmaverick.com/learning-terminated-merchan...
- 5y ago

fatnoah 5y ago

My last startup migrated from Verizon Terremark after the healthcare.gov fiasco several years ago. We also suffered from that massive outage and that was the final straw in migrating to AWS. At AWS, we built a few layers of redundant infrastructure with mulit-AZ availability within a region and then global availability across multiple regions. All this was done at roughly half the cost of the traditional hosting, even when including the additional person-hours required to maintain it on our end. Keeping our infra simple helped that work, and it's literally been years since an outage caused by any AWS issues, even though there have been several large AWS events.

zymhan 5y ago
Indeed, if you only deploy resources in us-east1, or any other single region, you're risking the occasional downtime. I'd wager that will still give you more uptime than a physically-hosted solution for the same cost.
- hinkley 5y ago
  Honestly, I have an app in production that isn't completely hardened against single zone outages. There was pressure to turn off some redundancy in our caching infra, and not every backend service we call is free of tenant affinity, so we could well lose at least 1/3rd of our customers in a single AZ failure in the wrong region, or have huge latency issues for all of our tenants based on high cache miss rates. Having written this, I'm going to ping our SME on the cache replication and remind him that since the last time he benchmarked it, we've upgraded to a newer generation of EC2 instances that has lower latency, and could he please run those numbers again.
- fatnoah 5y ago
  As I discovered many years ago when our infra was only in US-EAST1, failures were also easier to explain since many, many other companies would be offline as well. It made it more of an "Internet problem" than our own company's problem. For whatever reason, customers were far more likely to accept those kinds of outages.
hinkley 5y ago

kburman 5y ago

AWS or Google or any other reputable cloud provider are still far more better options then your local backup. Only way I see you losing your data is account getting locked.

hinkley 5y ago

Have you contacted them to see how things are going? Maybe a cheery note asking how the team is doing, sent right in the middle of an outage. Passive aggressive? As hell. Cathartic? Damn skippy.

whydoyoucare 5y ago

It reminds me of the old adage: "Two is one, one is none. Have a backup. Always."

deleted 5y ago

[deleted]