2 ms·

"Unless you're practicing continuous restoration to ensure that your backups are actually backing up what you need, the odds that you got everything critical ar

"Unless you're practicing continuous restoration to ensure that your backups are actually backing up what you need, the odds that you got everything critical are not good." "OK, but how about at least testing that that it's actually feasible to restore from that S3 bucket of yours?" These comments have been both off topic and unhelpful. I gave an example of S3 with object lock (or veam equivalents) that I liked - to be helpful to others exploring this question. Continuous type services are what this story is about - how things that are disconnected / not continuous may fare better in a hack / ransomware situation. Lecturing me that I need "continuous restoration" and or making random claims that I don't test restoration (huh?) is weird. If I was to give some feedback -> focus on one thing. If you don't like S3 with object lock, say that. If you want to add some comment about restorations from S3 with object lock being tricky for some reason, say it and explain it. If you have something about "continuous restoration" helping in malware hacks explain why. In many cases this can pass corrupted data over to your backup site if you are doing HA/DR work where continuous stuff is most common.

rectang 5y ago
The "one thing" I have been focusing on is testing backups to prove that they actually work as intended. The state of industry with regards to backups is wretched. If backups are done at all, they are often not tested — people just copy a bunch of files onto a RAID, or a USB drive, or an S3 bucket, or whatever, but never test restoration because that takes extra effort and may be inconvenient unless the system has been architected from day one to enable seamless restoration of live systems without obvious interruptions in service. Specifically with regards to ransomware hacks... The first thing that an attacker who gains root must do is corrupt production data. If there are no backups being created, or if the backups don't actually contain all critical data, then the hack is complete. If there are backups being performed, then the attacker must also hack the backup system sufficiently that restoration from available backup data is infeasible (or at least more costly than just paying the ransom). This adds an addiional layer of complexity to the attack. if restoration is being tested on an ongoing basis, then the attacker, in addition to corrupting the production data, must also spoof monitoring so that the the corruption of live and backup data doesn't get caught and interrupted before the corruption finishes. This adds yet another layer of complexity to the attack. A sufficiently competent and thorough attacker may overcome all these difficulties, but criminals are not always perfectly competent — they don't need to be because there are so many target organizations that are not competent either. It seems that your specific organization does thoroughly test backups to prove that restoration actually works as intended. Kudos! However, I wasn't evaluating your specific case — that would have been weird and inappropriate as you were just giving a (helpful!) example, not laying out a comprehensive system design and asking for review. My point was to add on the condition that in principle backed up data must be not only be 1) current and 2) not corrupted, but also 3) complete. And the way to prove completeness of backups is to test them, ideally on an ongoing basis.

rectang 5y ago

The "one thing" I have been focusing on is testing backups to prove that they actually work as intended. The state of industry with regards to backups is wretched. If backups are done at all, they are often not tested — people just copy a bunch of files onto a RAID, or a USB drive, or an S3 bucket, or whatever, but never test restoration because that takes extra effort and may be inconvenient unless the system has been architected from day one to enable seamless restoration of live systems without obvious interruptions in service. Specifically with regards to ransomware hacks... The first thing that an attacker who gains root must do is corrupt production data. If there are no backups being created, or if the backups don't actually contain all critical data, then the hack is complete. If there are backups being performed, then the attacker must also hack the backup system sufficiently that restoration from available backup data is infeasible (or at least more costly than just paying the ransom). This adds an addiional layer of complexity to the attack. if restoration is being tested on an ongoing basis, then the attacker, in addition to corrupting the production data, must also spoof monitoring so that the the corruption of live and backup data doesn't get caught and interrupted before the corruption finishes. This adds yet another layer of complexity to the attack. A sufficiently competent and thorough attacker may overcome all these difficulties, but criminals are not always perfectly competent — they don't need to be because there are so many target organizations that are not competent either. It seems that your specific organization does thoroughly test backups to prove that restoration actually works as intended. Kudos! However, I wasn't evaluating your specific case — that would have been weird and inappropriate as you were just giving a (helpful!) example, not laying out a comprehensive system design and asking for review. My point was to add on the condition that in principle backed up data must be not only be 1) current and 2) not corrupted, but also 3) complete. And the way to prove completeness of backups is to test them, ideally on an ongoing basis.