4 ms·

Honestly, this is one of the problems the cloud is great at solving. We keep things in seprate projects, and restore a backup from one production project (Kube

Honestly, this is one of the problems the cloud is great at solving. We keep things in seprate projects, and restore a backup from one production project (Kubernetes, Database servers, etc) to a special DR project we have set aside. The only step we do not do is update the front end DNS, and then run our tests after our Infrastrucutre as Code deployment is complete.

rhizome 5y ago
So a complete hot standby?
- Godel_unicode 5y ago
  Sounds like a DR test; spin up an entire new thing to make sure the backups worked, then turn it off. Costs a few hours of extra resources.
  - dspillett 5y ago
    I do something similar with my home mail server (not sure I'd go with self-hosting if I started fresh today, but it has run without issue other than maintenance for upgrades & such for well over a decade). I have a second copy running in a low spec VM. Every day it wipes itself down and restores the latest backup, I check regularly (a couple of times per week) to see that it is running and has copies of recent mail (that part could be more automated but I've never go round to it...). It wasn't setup for DR specifically, but it is on a separate network so if push came to shove I could ramp up its resources, change a few DNS settings, open up relevant bits of its firewalling, and have it take over, if the main server failed in a way it hadn't copied yet.
  - rhizome 5y ago
    Could be; it's ambiguous. But yeah, not updating DNS could be just declining to test operation in the wilds of outside traffic before tearing it all down. That said, in this way, maybe a periodic "DR" that actually replaces the current operations would be a helpful...well, not test, but..."resilience practice" maybe? It could be a new twist on continuous deployment: continuous recovery.
Johnny555 5y ago
Honestly, this is one of the problems the cloud is great at solving It's also one of the problems the cloud is great at causing. It's certainly possible to architect maintainable (and repeatable) infrastructure in the cloud. But it's just as easy (or easier) to deploy a mess of unorganized VM's that were launched and configured by whoever needed one.

rhizome 5y ago

So a complete hot standby?

Godel_unicode 5y ago
Sounds like a DR test; spin up an entire new thing to make sure the backups worked, then turn it off. Costs a few hours of extra resources.
- dspillett 5y ago
  I do something similar with my home mail server (not sure I'd go with self-hosting if I started fresh today, but it has run without issue other than maintenance for upgrades & such for well over a decade). I have a second copy running in a low spec VM. Every day it wipes itself down and restores the latest backup, I check regularly (a couple of times per week) to see that it is running and has copies of recent mail (that part could be more automated but I've never go round to it...). It wasn't setup for DR specifically, but it is on a separate network so if push came to shove I could ramp up its resources, change a few DNS settings, open up relevant bits of its firewalling, and have it take over, if the main server failed in a way it hadn't copied yet.
- rhizome 5y ago
  Could be; it's ambiguous. But yeah, not updating DNS could be just declining to test operation in the wilds of outside traffic before tearing it all down. That said, in this way, maybe a periodic "DR" that actually replaces the current operations would be a helpful...well, not test, but..."resilience practice" maybe? It could be a new twist on continuous deployment: continuous recovery.

Johnny555 5y ago

Honestly, this is one of the problems the cloud is great at solving It's also one of the problems the cloud is great at causing. It's certainly possible to architect maintainable (and repeatable) infrastructure in the cloud. But it's just as easy (or easier) to deploy a mess of unorganized VM's that were launched and configured by whoever needed one.