3 ms·

I have written a few documents outlining some risks and possible mitigations against AWS problems. It's not just "Russia nuked North Virginia". That's not by f

I have written a few documents outlining some risks and possible mitigations against AWS problems. It's not just "Russia nuked North Virginia". That's not by far the most likely failure scenario in my estimation. I consider "billing problem causes Amazon to close our account" or "trusted employee with top level access deletes everything in our account right down to Glacier storage" or "employee with access severely violates AWS T&Cs and gets us booted off" as being far more realistic threats, while effectively having the same consequences as a couple of ICBMs with MIRVs targeted to every datacenter in NV... In any of those cases, expecting to rehydrate from S3 isn't an option. Of course, nobody wants to incur the costs associated with "How could we keep business continuity without AWS?" because it gets very expensive very quickly for anything non trivial.

swyx 5y ago
interesting threat vector. would CEO having sole access to Cloudflare R2 clone satisfy at least data durability? since full continuity without AWS is probably impossible for most real life scenarios.
- brazzy 5y ago
  If the CEO has sole access to it, how do the backups get there?
  - AdamN 5y ago
    Having different read/write permissions is an important durability consideration. In this case the CEO has read privs (not write) and the archive utilities have write privs, not read. There are other problems of course, CEO being hit by a bus being the most obvious.
    - brazzy 5y ago
      > In this case the CEO has read privs (not write) and the archive utilities have write privs, not read. Not sure you could really make this work. The archive utilities would also need to be barred from overwriting or deleting backups. If that's automated, who configures it? And how do you do differential backups without read privileges?
      - AdamN 5y ago
        The archive utility, when run as the backup user, cannot read the file but it can read file metadata or have a database of backup state so it knows where to send differential backups. The archive utility, run as the recover user, can only read the file.
Dylan16807 5y ago
> Of course, nobody wants to incur the costs associated with "How could we keep business continuity without AWS?" because it gets very expensive very quickly for anything non trivial. I think that mostly depends on how much data you generate in a day. If sending backups out increases your bandwidth use by 5% then it's pretty easy to throw that into cold storage on google or azure or local or all three.
- bigiain 5y ago
  It's not just the data (that's an easy enough problem to solve). It's the infrastructure. Even if you've only got a reasonable simple platform, say some redundant EC2 app servers behind a load balancer and a multi AZ RDS database, with some S3 storage and a Cloudfront distribution serving static assets - you've probably also got Route53 DNS hosting, AKS ssl certs, deployment AMIs, CloudWatch monitoring/alerting, and a bunch of other "incidental" but effectively proprietary AWS stuff - because it's there. How do you get all that stood up "right now" in Azure or GCP or DigitalOcean or wherever, unless you've already put the time/effort into making that happen? How many "single points of failure" are locked inside your AWS account? (For my stuff, Route53 is the thing that keeps me up at night occasionally. If we lost access to our domains registered/hosted in AWS, we'd need to pick new domain names and update all out apps...)
  - Dylan16807 5y ago
    I guess it depends on how much you need "right now". It doesn't take crazy amounts of effort to set up app servers, a load balancer, a database, and S3-compatible storage somewhere else. If you had one person working on that two days a month you could keep a warm fallback system ready to go. That takes some effort to keep a map of what your cloud services are actually doing, which is a good idea anyway.

swyx 5y ago

interesting threat vector. would CEO having sole access to Cloudflare R2 clone satisfy at least data durability? since full continuity without AWS is probably impossible for most real life scenarios.

brazzy 5y ago
If the CEO has sole access to it, how do the backups get there?
- AdamN 5y ago
  Having different read/write permissions is an important durability consideration. In this case the CEO has read privs (not write) and the archive utilities have write privs, not read. There are other problems of course, CEO being hit by a bus being the most obvious.
  - brazzy 5y ago
    > In this case the CEO has read privs (not write) and the archive utilities have write privs, not read. Not sure you could really make this work. The archive utilities would also need to be barred from overwriting or deleting backups. If that's automated, who configures it? And how do you do differential backups without read privileges?
    - AdamN 5y ago
      The archive utility, when run as the backup user, cannot read the file but it can read file metadata or have a database of backup state so it knows where to send differential backups. The archive utility, run as the recover user, can only read the file.

Dylan16807 5y ago

> Of course, nobody wants to incur the costs associated with "How could we keep business continuity without AWS?" because it gets very expensive very quickly for anything non trivial. I think that mostly depends on how much data you generate in a day. If sending backups out increases your bandwidth use by 5% then it's pretty easy to throw that into cold storage on google or azure or local or all three.

bigiain 5y ago
It's not just the data (that's an easy enough problem to solve). It's the infrastructure. Even if you've only got a reasonable simple platform, say some redundant EC2 app servers behind a load balancer and a multi AZ RDS database, with some S3 storage and a Cloudfront distribution serving static assets - you've probably also got Route53 DNS hosting, AKS ssl certs, deployment AMIs, CloudWatch monitoring/alerting, and a bunch of other "incidental" but effectively proprietary AWS stuff - because it's there. How do you get all that stood up "right now" in Azure or GCP or DigitalOcean or wherever, unless you've already put the time/effort into making that happen? How many "single points of failure" are locked inside your AWS account? (For my stuff, Route53 is the thing that keeps me up at night occasionally. If we lost access to our domains registered/hosted in AWS, we'd need to pick new domain names and update all out apps...)
- Dylan16807 5y ago
  I guess it depends on how much you need "right now". It doesn't take crazy amounts of effort to set up app servers, a load balancer, a database, and S3-compatible storage somewhere else. If you had one person working on that two days a month you could keep a warm fallback system ready to go. That takes some effort to keep a map of what your cloud services are actually doing, which is a good idea anyway.