3 ms·
While as others already pointed out this story seems a little fishy, it serves well to reflect if something like this could in theory happen to your infrastruct
by jo909 11y ago
While as others already pointed out this story seems a little fishy, it serves well to reflect if something like this could in theory happen to your infrastructure.
Do you have your backup servers in the same configuration management software (ansible, puppet, ssh-for-loop etc) as the rest of the servers? One grave error (however unlikely) in your base configuration really can take down everything together in one fell swoop.
How "cold" are your backups? If the backup media are not physically disconnected and secured, you can most likely construct a scenario where the above, malware, a hacker or a rouge admin could destroy both the backups and the live data.
I will certainly suggest some additional safeguards for our backups.
- ansible 11y agoYep, that's what I hope everyone will be doing... thinking about their own backups and infrastructure. We have backups off-site on disconnected media, so that alone prevents the kind of accident we're talking about. We use btrfs send / receive to send OS images from the primary container host to the backup container host. The snapshots are read-only, so I'm fairly sure I can't just 'rm -rf' them, I'd have to actually 'btrfs subvolume delete foobar' them. I should try that though on one of the test servers...