8 ms·

Whenever I ask a CTO if they have a backup (or plan-B) they say we're on AWS, we backup there and they will never go down as a company. And then I ask them what

Whenever I ask a CTO if they have a backup (or plan-B) they say we're on AWS, we backup there and they will never go down as a company. And then I ask them what they do when their account gets shut (e.g. because they are selling something bad on Amazon and have the same phone number as the company account?) Or the instance some years ago where GCP closed because someone had wrongly classified image on their drive? You should have all you backups in a different location and terraform tested with a different cloud provider, otherwise you're risking the company. [Edit] Where I come from: That doesn't say anything about Hetzner, I have been with them for 20+ years, they have stopped individual servers in that time frame, but haven't cancelled my whole account.

diggan 2y ago
Another great question is "When did you last try to restore from a backup?" which usually is answered with "It's the built-in tooling, why would we assume it's broken?" or similar. Then fast-forward some months/years, and they try to restore from backups only to realize the backups never actually backed up what they cared about.
- KingOfCoders 2y ago
  Agreed, if you haven't tested your backups recently (daily, automatic best), you don't have backups. Several of my clients (CTO Coaching) had problems in the past because they restored backups and where finding they were not complete (for various reasons).
  - w8whut 2y ago
    daily test restore is infeasible for anything but toy projects. You should periodically test your restore procedures, but its incredibly costly and time consuming for sizeable platforms. Its just not that easy to restore a 10+TB backup for example, and thats a _tiny_ backup size for a b2c product. they can easily go into the hundreds of TB, depending on your platform. and i might add: i vividly remember gitlabs article how they have had automated backups and test restores for years, but when they actually needed them... it turned out some data wasn't part of it after all. just because youre testing your restore procedure doesnt mean you've actually accomplished anything.
    - rodgerd 2y ago
      > but its incredibly costly and time consuming for sizeable platforms. If your restores are too time consuming to test regularly, they sure as shit aren't going to be useful in a disaster.
      - usrnm 2y ago
        Why is that? Hours or even days of downtime are still better than just losing all data. It's a simple cost-benefit analysis, and it's ok to pick different trade-offs depending on your use case
        rodgerd 2y ago
        I have spent most of my career in newspaper publishing and banking. A newspaper that doesn't publish for a few days might recover. A bank that drops off the Swift network for days isn't a bank any more.
        cyberax 2y ago
        Russia and Iran would like to have a few words with you. Banks regularly close for multiple days for bank holidays. Unscheduled downtime is a somewhat different story, though. Luckily, the traditional SWIFT/banking infrastructure is so negligible these days, my phone can host a classic banking infrastructure for an entire small country.
        deleted 2y ago
        [deleted]
        com 2y ago
        This is a nonsense response. A bank exists not as an isolated entitity, but as a node in a local, regional and global network of transactions. Your phone as a “classic banking infrastructure” (nonsense phrase) can’t do credit card acquiring or realtime transactions because it’s not connected to the payment rails, transaction switches and so on (like SWIFT but all of the other less global ones run by central banks and private entities). In developed societies, instant settlement for bank-bank transfers is the norm, and cash flow is dependent on that. Russia and Iran pay about 2-5% for above-board (non-sanctioned) cross border transactions due to their extra costs of not being in SWIFT and USD sanctions, and between 20-50% where physical middle-people are needed to move pallets of USD.
- amyames 2y ago
  aws-cli will sync your s3 buckets to a local system. I’m doing that to linux, and then the Linux box is furthermore backed up with nakivo. Not my favorite but the price was okay and I can run the whole director on Linux, unlike all their other competitors. [veeam’s next major release 13 or 14 should do this in the next year or so too.] While nakivo backs up s3 buckets, NFS shares, and local file servers… to your point, I don’t trust it (or any other backup software I can’t extract and unpack the resulting backup by hand) as far as I can throw it. So I rsync or mirror it to a local Linux box with aws-cli and then back THAT up. I think you can do all this with windows stuff too but I don’t know it that well Additionally you can take servers that are linux vps’es and do the reverse: mirror THEIR content to an s3 bucket. You can also run minio open source/free on your fileserver and set up s3 to s3 sync. Cloudflare for example will ingest and replicate your minio server automatically and you can firewall it all off to their address ranges. It’s not free but it actually prices out favorably compared to veeam and nakivo if that’s all you need backed up.
- ambicapter 2y ago
  One thing I've never figured out is, what is the difference between backups and replication? And, does restoring from backups always mean losing more _recent_ data than replication?
  - rolph 2y ago
    replication is a snapshot of everything, at the time of file access. backup is a replacement of specified files required by a system recovery procedure. it may be a total image, or a collection of config, and dat files, that are daily bootup settings,
  - justusthane 2y ago
    Replication isn't a backup, because if you accidentally delete a file, and that deletion is immediately replicated, then you can't get that file back. Backups are a specific point in time.
    - mavhc 2y ago
      Backup is a useless word, it's too overloaded Are snapshots backups? snapshots on raid? snapshots on replicated disks?
      - ianburrell 2y ago
        Backup describes what they are for. There are many ways to do backups. The main thing is that backups are archive stored away from danger. There are different kinds of danger and different need for protection. Snapshots can be backups depending on where they are stored usually not if stored locally. For example, RDS snapshot is backup for database going down but not account being deleted or region destroyed. Generally, snapshots are way to make backups to more durable medium.
      - andrewaylett 2y ago
        A backup is something that will functionally replace the original should the original fail, regardless of how the original failed. For data, this means that the restore process is part of the backup. Snapshots are not backups. Snapshots on RAID are not backups. Snapshots on replicated disks are probably backups, so long as the disks being replicated to are not inside the same case/building/city/continent (pick your risk suitably) and you're not able to delete the snapshots from the machine hosting the originals. The second SIM in my phone provides a backup for my primary service provider, so long as I keep it activated. The torch in my pocket is a backup for the lighting in my house, so long as I keep it charged. My data in tarsnap is a backup, so long as I'm able to restore it. Which means data in tarsnap isn't a complete backup on its own: unless I'm able to recover the encryption key, I don't actually have a backup.
- sidewndr46 2y ago
  my dad told me about this customer that had a server that made automatic backups each Sunday night. The backup script would backup all the data then eject the tape so the manager could put it in the vault and rotate in the other one from the vault. When the hard drive failed, they restored the customer to the latest backup. Which was the tape still sitting in the tape drive in the server. It was from the first Sunday night after the system was installed years ago
  - Terr_ 2y ago
    I'm confused, it sounds like you're saying the same tape was being ejected every week and then reinserted without any rotation. But in that case, shouldn't the weekly backup process have failed because the tape was full? Was nobody getting those alerts? Or do you mean the backup process was fine, but they restored from the wrong media, a very old tape that was about to be overwritten, instead of retrieving the one with last-week's copy?
    - ender341341 2y ago
      I read it as they were saying the manual part of the process never happened, so the backup from the first week was just sitting ejected forever and they had no alarming to notify them that the new backup failed to write to tape.
    - sidewndr46 2y ago
      no, the first Sunday night after the backup process completed. It ejected the tape. It sat there for years until someone realized they needed to restore from the most recent backup. Since a new tape was never inserted, the backup was from years ago.
      - deleted 2y ago
        [deleted]
      - cutemonster 2y ago
        What happened afterwards? Was that manager still at the company, at that time?
- remram 2y ago
  This famously happened at GitLab: https://about.gitlab.com/blog/2017/02/01/gitlab-dot-com-database-incident/ https://about.gitlab.com/blog/2017/02/01/gitlab-dot-com-data... > Regular backups seem to also only be taken once per 24 hours, though team-member-1 has not yet been able to figure out where they are stored. According to team-member-2 these don’t appear to be working, producing files only a few bytes in size.
- dawnerd 2y ago
  In addition to making sure it works, you should make sure you know how to deal with restoring. Sure running a command is easy but what about spinning up new infra? What if it’s corrupted? What if the one person that knows the setup is gone or asleep? Mainly a problem for smaller teams that don’t have the redundancy or resources - they really need to make sure there’s at least docs on how stuff was setup. Reminds me I need to do my yearly checkup too.
- bazzargh 2y ago
  A fun one like that, a few years back we had some code using dynamodb that used the automatic point-in-time backups. I asked if it had been tested, need you guess the reply? Of course it turns out that the restore can only happen to a _new database name_ not the original, and the code had in multiple places hardcoded the assumption of what the db was called. So restoring also involved patching the code and rolling that out; you can't "roll back" because to roll back the db the code must roll forward.
  - banku_brougham 2y ago
    'Roll-middle-out'
  - jamesfinlayson 2y ago
    Ugh, very annoying limitation. I've written scripts to create dummy tables, restore backups and sync across to existing tables twice now.
- kstrauser 2y ago
  We've avoided that in various shops by making backups/restores part of regular maintenance processes. How do we upgrade the database? By stopping it, backing it up, restoring that to the new server, pointing all code at the new DB, then turning off the old server. As with code deployment, it's not so scary when it's something you do so frequently that it's just a little script you run.
  - diggan 2y ago
    > it's not so scary when it's something you do so frequently Yeah, I've found this to be the trick for ongoing hassle-free maintenance too. Make tearing stuff down and up frequent enough and you'll feel confident and safe when you're required to do so to recover from something. Scariest are applications/services/servers that has been running for years but never restarted nor ever restored. Those scare me.
    - kstrauser 2y ago
      Cattle, not kittens. My favorite thing about deploying containerized apps? They’re completely fungible and I never have to care about an individual instance. Oh, it hung due to some weird network interaction? Spawn a new one, then come back to see what went wrong with this one before you kill it.
- belter 2y ago
  "Amateurs backup. Professionals restore."
- auspex 2y ago
  You have to test restoration as part of SOC 2, so most companies with real customers do it at least once a year.
endgame 2y ago
https://cloud.google.com/blog/products/infrastructure/details-of-google-cloud-gcve-incident https://cloud.google.com/blog/products/infrastructure/detail... Google Cloud accidentally wiped an Australian super[annuation] (pension) fund's entire cloud deployment earlier this year. I think that if you really want durable backups, they have to be reducible to object storage and put in someone else's cloud.
- milesward 2y ago
  ... not quite. I worked directly with the folks involved on getting more RCA details public. This customer used a single product on GCP, a specific type of VMware hosting, and the "subscription" to that product failed, which turned those resources off. It's more like turning off all their VM's, rather than deleting their entire account, identities, access structures, etc.
  - buran77 2y ago
    The reporting on that was a bid muddy with Google and Unisuper officially saying different things in different places. Regardless, calling it "more like turning off all their VM's" sounds like heavily downplaying the reality. The downtime alone confirms it was way more than that. From their joint statement [0]: > when the deletion of UniSuper’s Private Cloud subscription occurred, it caused deletion across both of these geographies. > an extensive recovery of our Private Cloud which includes hundreds of virtual machines, databases and applications. > UniSuper had backups in place with an additional service provider. These backups have minimised data loss Strangely enough on this last point a Google blog post [1] says: > This incident did not impact: The customer’s data backups stored in Google Cloud Storage (GCS) in the same region. [0] https://www.unisuper.com.au/about-us/media-centre/2024/a-joint-statement-from-unisuper-and-google-cloud https://www.unisuper.com.au/about-us/media-centre/2024/a-joi... [1] https://cloud.google.com/blog/products/infrastructure/details-of-google-cloud-gcve-incident https://cloud.google.com/blog/products/infrastructure/detail...
- throwaway2037 2y ago
  Thank you to share that blog post. That blog post specifically mentioned that no data was lost. I am confused by your comment about durable backups. Deeper question: Do people think on-prem backups are more reliable than cloud? I would say for 95% of orgs: no.
hypeatei 2y ago
I agree about data backups but replicating your setup in another cloud provider is: 1) Expensive 2) Not straightforward, e.g. is there a 1:1 setup in another cloud for your system? 3) Likely to go untested and be useless when you need it most
- benterix 2y ago
  Fully agree, that's why you need to think well first and come up with a compromise that you are willing to accept. Periodic testing of your DR procedures is non-negotiable but fortunately it's usually much simpler for smaller startups than for larger orgs.
- KingOfCoders 2y ago
  I do think (1.) depends on your company size, and business model. For most, it's cheap, e.g. https://rsync.net/pricing.html https://rsync.net/pricing.html That said I was once a CTO for a company with 10 photo studios and we had a large amount of new (raw, DSLR) photos per minute, so cost was an issue and also upload speed for offsite backups.
- crote 2y ago
  The best part of cloud providers is that short-term VMs are relatively cheap to deploy. You don't need a full active-active failover setup, you just need to design your infrastructure in a cloud-agnostic way and test the deployment scripts a few times a year. The most expensive part is going to be maintaining an up-to-date offsite data backup. Running a few VMs for a handful of hours is basically free.
  - kstrauser 2y ago
    > you just need to design your infrastructure in a cloud-agnostic way But that's one helluva "just", and also means that you can't use the platform-specific features that make life easier. In practice that's probably way more expensive than spinning some testing VPSes up and down.
    - crote 2y ago
no_wizard 2y ago
When should one start doing this though, in a companies life cycle? What is the most reasonable point that meets the criteria of 'as soon as possible'? Because I imagine out of the gate doing this could be a net negative, not a net positive. On the other hand, I'm not sufficiently well versed enough on the absolute latest devops techniques that may make this whole thing trivial, but I thought all the major cloud providers had just enough quirks in their Terraform support you can't write once standup / deploy anywhere
- marcosdumay 2y ago
  You should have tested backups by the time you have something running in production. It's very easy to do if you don't do the absolute latest devops techniques.
  - no_wizard 2y ago
    Really thinking about testing the whole “we could migrate to any cloud on a moments notice” idea
tracerbulletx 2y ago
There should just be a legal duty placed on cloud providers to not do this. Nobody would expect you to hold a second redundant commercial lease for your offices or retail location.
- patmcc 2y ago
  I think this is a tough problem, partly due to the post-paid nature of most cloud services, partly due to the impact to other customers. If you had a bunch of retailers in a shared space (like a market), and one of them was setting off fireworks, using all the power/water in the space, and scaring away customers, I'd expect them to get kicked out pretty quickly. Now it may be that this is a false positive, I'm sure they happen, but in the case where it's a legitimate bad actor that is actively harming both the company and other customers on those servers, what's the course of action the company should take?
- gruez 2y ago
  Shouldn't this be covered under standard tort law?
- kevincox 2y ago
  This isn't a great example because buildings do have accidents like fires and floods. If you need business continuity you do plan on having multiple working locations. Of course an accident is different than just randomly terminating service.
hmmm-i-wonder 2y ago
They've gone the route of multiple AWS accounts in my company to avoid the issue they introduced with horrible planning. First they wanted us out of on-premise, and told us costs wouldn't matter. Then they wanted us to be 'cloud agnostic', but when given deadlines changed to 'get it working in AWS ASAP, doesn't matter the tech debt' Now they're freaking out about AWS costs, and we're back to juggling 'cloud agnostic' and 'reduce cost to serve in all clouds' priorities on top of features and maintenance, both of which are 10x slower due to tech debt and the plethora of bugs. I really need to find a new job soon. Its insane how badly the execs and upper management are running this company. Every day is a knee jerk reaction from someone so detached from the reality of things or with so little understanding how it works, they do nothing but add process problems that barely address the issues they think they're solving.
- sevensor 2y ago
  I’d ask, “have we worked together?” since this is a spot on description of my former employer, except it’s probably a spot on description of thousands of mid sized companies.
  - unethical_ban 2y ago
    Same! Some execs get excited about reducing capital expenses for a data center and the teams that manage it. Some CTO gets excited about the flexibility and some legitimate benefits of cloud. But it ends up costing a shitton of money to switch paradigms completely, and they don't switch paradigms completely for a number of years: If you're just migrating servers to ec2/vpc, you're doing cloud wrong. Of course, there is the idea of cloud agnostic, or even multi region, which seems a challenge for most places. At least with terraform, it is theoretically easier to swing configurations over to a different host.
    - ryandrake 2y ago
      At many places I've worked, there are essentially zero checks-and-balances between "Exec gets randomly excited about X" and "X becomes a mandate, with staffing, budget, and deadline." No technical vetting, feedback loop, sometimes no apparent coordination with other execs (and their random ideas). It's just: "Mike is excited about Cloud. -> We are now doing Cloud." Later, Mike gets excited about something else, and the entire team moves over to something else. "Mike is excited about AI. -> We are now doing AI."
deleted 2y ago
[deleted]
bambax 2y ago
Of course, never put all your eggs in the same basket. Have a different registrar as well, and maybe a different CDN ready to go at a moment's notice.
rodgerd 2y ago
This is why the primary bank regulator in Australia (APRA) have insistent that banks meet their CPS230 obligations by being multicloud. There's a lot of push back on it (especially from AWS), but it's a significant risk if you're leasing all your infra.
486sx33 2y ago
I run daily backups of our entire GSuite domain to a local RAID 5 device Everyone thinks I’m crazy
evoke4908 2y ago
My CEO has been letting the AWS bill go unpaid, apparently not understanding that our entire business and all of our IP will simply vanish if our S3 bucket gets deleted. Zero backups of any kind I manually pulled a backup of everything but jeez, not good.
- quectophoton 2y ago
  Hopefully your warnings to them are in writing, and you have enough to CYA just in case.

diggan 2y ago

Another great question is "When did you last try to restore from a backup?" which usually is answered with "It's the built-in tooling, why would we assume it's broken?" or similar. Then fast-forward some months/years, and they try to restore from backups only to realize the backups never actually backed up what they cared about.

KingOfCoders 2y ago
Agreed, if you haven't tested your backups recently (daily, automatic best), you don't have backups. Several of my clients (CTO Coaching) had problems in the past because they restored backups and where finding they were not complete (for various reasons).
- w8whut 2y ago
  daily test restore is infeasible for anything but toy projects. You should periodically test your restore procedures, but its incredibly costly and time consuming for sizeable platforms. Its just not that easy to restore a 10+TB backup for example, and thats a _tiny_ backup size for a b2c product. they can easily go into the hundreds of TB, depending on your platform. and i might add: i vividly remember gitlabs article how they have had automated backups and test restores for years, but when they actually needed them... it turned out some data wasn't part of it after all. just because youre testing your restore procedure doesnt mean you've actually accomplished anything.
  - rodgerd 2y ago
    > but its incredibly costly and time consuming for sizeable platforms. If your restores are too time consuming to test regularly, they sure as shit aren't going to be useful in a disaster.
    - usrnm 2y ago
      Why is that? Hours or even days of downtime are still better than just losing all data. It's a simple cost-benefit analysis, and it's ok to pick different trade-offs depending on your use case
      - rodgerd 2y ago
        I have spent most of my career in newspaper publishing and banking. A newspaper that doesn't publish for a few days might recover. A bank that drops off the Swift network for days isn't a bank any more.
        cyberax 2y ago
        Russia and Iran would like to have a few words with you. Banks regularly close for multiple days for bank holidays. Unscheduled downtime is a somewhat different story, though. Luckily, the traditional SWIFT/banking infrastructure is so negligible these days, my phone can host a classic banking infrastructure for an entire small country.
        deleted 2y ago
        [deleted]
        com 2y ago
        This is a nonsense response. A bank exists not as an isolated entitity, but as a node in a local, regional and global network of transactions. Your phone as a “classic banking infrastructure” (nonsense phrase) can’t do credit card acquiring or realtime transactions because it’s not connected to the payment rails, transaction switches and so on (like SWIFT but all of the other less global ones run by central banks and private entities). In developed societies, instant settlement for bank-bank transfers is the norm, and cash flow is dependent on that. Russia and Iran pay about 2-5% for above-board (non-sanctioned) cross border transactions due to their extra costs of not being in SWIFT and USD sanctions, and between 20-50% where physical middle-people are needed to move pallets of USD.
amyames 2y ago
aws-cli will sync your s3 buckets to a local system. I’m doing that to linux, and then the Linux box is furthermore backed up with nakivo. Not my favorite but the price was okay and I can run the whole director on Linux, unlike all their other competitors. [veeam’s next major release 13 or 14 should do this in the next year or so too.] While nakivo backs up s3 buckets, NFS shares, and local file servers… to your point, I don’t trust it (or any other backup software I can’t extract and unpack the resulting backup by hand) as far as I can throw it. So I rsync or mirror it to a local Linux box with aws-cli and then back THAT up. I think you can do all this with windows stuff too but I don’t know it that well Additionally you can take servers that are linux vps’es and do the reverse: mirror THEIR content to an s3 bucket. You can also run minio open source/free on your fileserver and set up s3 to s3 sync. Cloudflare for example will ingest and replicate your minio server automatically and you can firewall it all off to their address ranges. It’s not free but it actually prices out favorably compared to veeam and nakivo if that’s all you need backed up.
ambicapter 2y ago
One thing I've never figured out is, what is the difference between backups and replication? And, does restoring from backups always mean losing more _recent_ data than replication?
- rolph 2y ago
  replication is a snapshot of everything, at the time of file access. backup is a replacement of specified files required by a system recovery procedure. it may be a total image, or a collection of config, and dat files, that are daily bootup settings,
- justusthane 2y ago
  Replication isn't a backup, because if you accidentally delete a file, and that deletion is immediately replicated, then you can't get that file back. Backups are a specific point in time.
  - mavhc 2y ago
    Backup is a useless word, it's too overloaded Are snapshots backups? snapshots on raid? snapshots on replicated disks?
    - ianburrell 2y ago
      Backup describes what they are for. There are many ways to do backups. The main thing is that backups are archive stored away from danger. There are different kinds of danger and different need for protection. Snapshots can be backups depending on where they are stored usually not if stored locally. For example, RDS snapshot is backup for database going down but not account being deleted or region destroyed. Generally, snapshots are way to make backups to more durable medium.
    - andrewaylett 2y ago
      A backup is something that will functionally replace the original should the original fail, regardless of how the original failed. For data, this means that the restore process is part of the backup. Snapshots are not backups. Snapshots on RAID are not backups. Snapshots on replicated disks are probably backups, so long as the disks being replicated to are not inside the same case/building/city/continent (pick your risk suitably) and you're not able to delete the snapshots from the machine hosting the originals. The second SIM in my phone provides a backup for my primary service provider, so long as I keep it activated. The torch in my pocket is a backup for the lighting in my house, so long as I keep it charged. My data in tarsnap is a backup, so long as I'm able to restore it. Which means data in tarsnap isn't a complete backup on its own: unless I'm able to recover the encryption key, I don't actually have a backup.
sidewndr46 2y ago
my dad told me about this customer that had a server that made automatic backups each Sunday night. The backup script would backup all the data then eject the tape so the manager could put it in the vault and rotate in the other one from the vault. When the hard drive failed, they restored the customer to the latest backup. Which was the tape still sitting in the tape drive in the server. It was from the first Sunday night after the system was installed years ago
- Terr_ 2y ago
  I'm confused, it sounds like you're saying the same tape was being ejected every week and then reinserted without any rotation. But in that case, shouldn't the weekly backup process have failed because the tape was full? Was nobody getting those alerts? Or do you mean the backup process was fine, but they restored from the wrong media, a very old tape that was about to be overwritten, instead of retrieving the one with last-week's copy?
  - ender341341 2y ago
    I read it as they were saying the manual part of the process never happened, so the backup from the first week was just sitting ejected forever and they had no alarming to notify them that the new backup failed to write to tape.
  - sidewndr46 2y ago
    no, the first Sunday night after the backup process completed. It ejected the tape. It sat there for years until someone realized they needed to restore from the most recent backup. Since a new tape was never inserted, the backup was from years ago.
    - deleted 2y ago
      [deleted]
    - cutemonster 2y ago
      What happened afterwards? Was that manager still at the company, at that time?
remram 2y ago
This famously happened at GitLab: https://about.gitlab.com/blog/2017/02/01/gitlab-dot-com-database-incident/ https://about.gitlab.com/blog/2017/02/01/gitlab-dot-com-data... > Regular backups seem to also only be taken once per 24 hours, though team-member-1 has not yet been able to figure out where they are stored. According to team-member-2 these don’t appear to be working, producing files only a few bytes in size.
dawnerd 2y ago
In addition to making sure it works, you should make sure you know how to deal with restoring. Sure running a command is easy but what about spinning up new infra? What if it’s corrupted? What if the one person that knows the setup is gone or asleep? Mainly a problem for smaller teams that don’t have the redundancy or resources - they really need to make sure there’s at least docs on how stuff was setup. Reminds me I need to do my yearly checkup too.
bazzargh 2y ago
A fun one like that, a few years back we had some code using dynamodb that used the automatic point-in-time backups. I asked if it had been tested, need you guess the reply? Of course it turns out that the restore can only happen to a _new database name_ not the original, and the code had in multiple places hardcoded the assumption of what the db was called. So restoring also involved patching the code and rolling that out; you can't "roll back" because to roll back the db the code must roll forward.
- banku_brougham 2y ago
  'Roll-middle-out'
- jamesfinlayson 2y ago
  Ugh, very annoying limitation. I've written scripts to create dummy tables, restore backups and sync across to existing tables twice now.
kstrauser 2y ago
We've avoided that in various shops by making backups/restores part of regular maintenance processes. How do we upgrade the database? By stopping it, backing it up, restoring that to the new server, pointing all code at the new DB, then turning off the old server. As with code deployment, it's not so scary when it's something you do so frequently that it's just a little script you run.
- diggan 2y ago
  > it's not so scary when it's something you do so frequently Yeah, I've found this to be the trick for ongoing hassle-free maintenance too. Make tearing stuff down and up frequent enough and you'll feel confident and safe when you're required to do so to recover from something. Scariest are applications/services/servers that has been running for years but never restarted nor ever restored. Those scare me.
  - kstrauser 2y ago
    Cattle, not kittens. My favorite thing about deploying containerized apps? They’re completely fungible and I never have to care about an individual instance. Oh, it hung due to some weird network interaction? Spawn a new one, then come back to see what went wrong with this one before you kill it.
belter 2y ago
"Amateurs backup. Professionals restore."
auspex 2y ago
You have to test restoration as part of SOC 2, so most companies with real customers do it at least once a year.

endgame 2y ago

https://cloud.google.com/blog/products/infrastructure/details-of-google-cloud-gcve-incident https://cloud.google.com/blog/products/infrastructure/detail... Google Cloud accidentally wiped an Australian super[annuation] (pension) fund's entire cloud deployment earlier this year. I think that if you really want durable backups, they have to be reducible to object storage and put in someone else's cloud.

milesward 2y ago
... not quite. I worked directly with the folks involved on getting more RCA details public. This customer used a single product on GCP, a specific type of VMware hosting, and the "subscription" to that product failed, which turned those resources off. It's more like turning off all their VM's, rather than deleting their entire account, identities, access structures, etc.
- buran77 2y ago
  The reporting on that was a bid muddy with Google and Unisuper officially saying different things in different places. Regardless, calling it "more like turning off all their VM's" sounds like heavily downplaying the reality. The downtime alone confirms it was way more than that. From their joint statement [0]: > when the deletion of UniSuper’s Private Cloud subscription occurred, it caused deletion across both of these geographies. > an extensive recovery of our Private Cloud which includes hundreds of virtual machines, databases and applications. > UniSuper had backups in place with an additional service provider. These backups have minimised data loss Strangely enough on this last point a Google blog post [1] says: > This incident did not impact: The customer’s data backups stored in Google Cloud Storage (GCS) in the same region. [0] https://www.unisuper.com.au/about-us/media-centre/2024/a-joint-statement-from-unisuper-and-google-cloud https://www.unisuper.com.au/about-us/media-centre/2024/a-joi... [1] https://cloud.google.com/blog/products/infrastructure/details-of-google-cloud-gcve-incident https://cloud.google.com/blog/products/infrastructure/detail...
throwaway2037 2y ago
Thank you to share that blog post. That blog post specifically mentioned that no data was lost. I am confused by your comment about durable backups. Deeper question: Do people think on-prem backups are more reliable than cloud? I would say for 95% of orgs: no.

hypeatei 2y ago

I agree about data backups but replicating your setup in another cloud provider is: 1) Expensive 2) Not straightforward, e.g. is there a 1:1 setup in another cloud for your system? 3) Likely to go untested and be useless when you need it most

benterix 2y ago
Fully agree, that's why you need to think well first and come up with a compromise that you are willing to accept. Periodic testing of your DR procedures is non-negotiable but fortunately it's usually much simpler for smaller startups than for larger orgs.
KingOfCoders 2y ago
I do think (1.) depends on your company size, and business model. For most, it's cheap, e.g. https://rsync.net/pricing.html https://rsync.net/pricing.html That said I was once a CTO for a company with 10 photo studios and we had a large amount of new (raw, DSLR) photos per minute, so cost was an issue and also upload speed for offsite backups.
crote 2y ago
The best part of cloud providers is that short-term VMs are relatively cheap to deploy. You don't need a full active-active failover setup, you just need to design your infrastructure in a cloud-agnostic way and test the deployment scripts a few times a year. The most expensive part is going to be maintaining an up-to-date offsite data backup. Running a few VMs for a handful of hours is basically free.
- kstrauser 2y ago
  > you just need to design your infrastructure in a cloud-agnostic way But that's one helluva "just", and also means that you can't use the platform-specific features that make life easier. In practice that's probably way more expensive than spinning some testing VPSes up and down.
  - crote 2y ago

no_wizard 2y ago

When should one start doing this though, in a companies life cycle? What is the most reasonable point that meets the criteria of 'as soon as possible'? Because I imagine out of the gate doing this could be a net negative, not a net positive. On the other hand, I'm not sufficiently well versed enough on the absolute latest devops techniques that may make this whole thing trivial, but I thought all the major cloud providers had just enough quirks in their Terraform support you can't write once standup / deploy anywhere

marcosdumay 2y ago
You should have tested backups by the time you have something running in production. It's very easy to do if you don't do the absolute latest devops techniques.
- no_wizard 2y ago
  Really thinking about testing the whole “we could migrate to any cloud on a moments notice” idea

tracerbulletx 2y ago

There should just be a legal duty placed on cloud providers to not do this. Nobody would expect you to hold a second redundant commercial lease for your offices or retail location.

patmcc 2y ago
I think this is a tough problem, partly due to the post-paid nature of most cloud services, partly due to the impact to other customers. If you had a bunch of retailers in a shared space (like a market), and one of them was setting off fireworks, using all the power/water in the space, and scaring away customers, I'd expect them to get kicked out pretty quickly. Now it may be that this is a false positive, I'm sure they happen, but in the case where it's a legitimate bad actor that is actively harming both the company and other customers on those servers, what's the course of action the company should take?
gruez 2y ago
Shouldn't this be covered under standard tort law?
kevincox 2y ago
This isn't a great example because buildings do have accidents like fires and floods. If you need business continuity you do plan on having multiple working locations. Of course an accident is different than just randomly terminating service.

hmmm-i-wonder 2y ago

They've gone the route of multiple AWS accounts in my company to avoid the issue they introduced with horrible planning. First they wanted us out of on-premise, and told us costs wouldn't matter. Then they wanted us to be 'cloud agnostic', but when given deadlines changed to 'get it working in AWS ASAP, doesn't matter the tech debt' Now they're freaking out about AWS costs, and we're back to juggling 'cloud agnostic' and 'reduce cost to serve in all clouds' priorities on top of features and maintenance, both of which are 10x slower due to tech debt and the plethora of bugs. I really need to find a new job soon. Its insane how badly the execs and upper management are running this company. Every day is a knee jerk reaction from someone so detached from the reality of things or with so little understanding how it works, they do nothing but add process problems that barely address the issues they think they're solving.

sevensor 2y ago
I’d ask, “have we worked together?” since this is a spot on description of my former employer, except it’s probably a spot on description of thousands of mid sized companies.
- unethical_ban 2y ago
  Same! Some execs get excited about reducing capital expenses for a data center and the teams that manage it. Some CTO gets excited about the flexibility and some legitimate benefits of cloud. But it ends up costing a shitton of money to switch paradigms completely, and they don't switch paradigms completely for a number of years: If you're just migrating servers to ec2/vpc, you're doing cloud wrong. Of course, there is the idea of cloud agnostic, or even multi region, which seems a challenge for most places. At least with terraform, it is theoretically easier to swing configurations over to a different host.
  - ryandrake 2y ago
    At many places I've worked, there are essentially zero checks-and-balances between "Exec gets randomly excited about X" and "X becomes a mandate, with staffing, budget, and deadline." No technical vetting, feedback loop, sometimes no apparent coordination with other execs (and their random ideas). It's just: "Mike is excited about Cloud. -> We are now doing Cloud." Later, Mike gets excited about something else, and the entire team moves over to something else. "Mike is excited about AI. -> We are now doing AI."

deleted 2y ago

[deleted]

bambax 2y ago

Of course, never put all your eggs in the same basket. Have a different registrar as well, and maybe a different CDN ready to go at a moment's notice.

rodgerd 2y ago

This is why the primary bank regulator in Australia (APRA) have insistent that banks meet their CPS230 obligations by being multicloud. There's a lot of push back on it (especially from AWS), but it's a significant risk if you're leasing all your infra.

486sx33 2y ago

I run daily backups of our entire GSuite domain to a local RAID 5 device Everyone thinks I’m crazy

evoke4908 2y ago

My CEO has been letting the AWS bill go unpaid, apparently not understanding that our entire business and all of our IP will simply vanish if our S3 bucket gets deleted. Zero backups of any kind I manually pulled a backup of everything but jeez, not good.

quectophoton 2y ago
Hopefully your warnings to them are in writing, and you have enough to CYA just in case.