4 ms·
A generally safe backup process looks like this: Production has no access to backup. Backup has read only access to production. Backup writes are append and
by johngalt 7y ago
A generally safe backup process looks like this:
Production has no access to backup.
Backup has read only access to production.
Backup writes are append and not overwrites.
Deletes/archival are governed by a retention process.
- Damogran6 7y agoMonitor dedupe statistics. If they go pear shaped, Copy N is much different than Copy N+1.
- lifeisstillgood 7y agoBy good and easy to do point- something as simple as size will help but a diff between backups would tell much - i had not thought of that - then again it comes for free in something like tarsnap
- pstuart 7y agoA solid list. It's nice to have a restore step in there too, so one has both a validation that the backup is usable, and gives a "playground" where one can have a safe day old environment for testing/training/whatnot against production data.
- __jal 7y agoI don't consider that nice-to-have. That is essential. If you do not regularly test restoring, you do not know if you have backups. I normally overstate this just a little for effect, "If you don't test your backups you don't have backups", but that isn't precisely true. And yes, it frequently makes sense to base test/fixture data and whatnot on backup data flows, but unless you don't deal with any PII, you're hopefully not using that data without a sanitization step.
- JustSomeNobody 7y agoDo you restore to prod? Or to Non-Prod and compare to prod?
- johngalt 7y agoThank you. To anyone viewing my answer as a guide: a backup by itself is not a continuity plan. There are other important areas such as setting recovery metrics (RPO/RTO) and testing restores against those metrics. The world is rife with stories of unreadable backups and/or restorations that took days instead of hours.
- gowld 7y agoAlso, backups must include a physical process of moving a copy of backups to an airgapped secondary system (a human ejecting disks/CDs/tapes and carrying them to another storage container), so that it's impossible for an attacker to compromise the backups via the same software exploit that corrupts the primary data.
- ipython 7y agoOne absolutely diabolical mechanism that was used (at least 5 years ago when this scourge of ransomware started to rear its ugly head) goes something like this: 1. Gain access to change the code on the front-end web servers (usually PHP) 2. Change the database access layer to transparently encrypt data being written to the database, and decrypt data being read from the database. The key would be loaded into memory by curl'ing an attacker-controlled website at startup. 3. Wait 30 days 4. Notify the company that they're compromised, turn off the attacker-controlled key service, and restart the web front end Now step (3) ensures that most data in the database has been re-written, and if your backups are dumps of the production database, you now have a month of encrypted backups that you can't read... If you're lucky, you may have a month-old backup to restore from; if you're unlucky, you rotate every 30 days.
- mgleason_3 7y agoExactly. It’s possible they are screwed.
- londons_explore 7y agoHas this actually happened? Implementing zero downtime transparent encryption on any moderately complex codebase above the database layer sounds like a big engineering challenge. Even moreso if you're trying to do it without the sysadmin noticing. Even moreso if there is no persistence for the encryption key so every user request ends up having to fire a request to the attackers server - that'll kill performance. I doubt many attackers would attempt it.
- ipython 7y agoIf I’ve learned anything in decades of working in infosec, it’s to never underestimate the ingenuity of a determined attacker. Read through any modern multi stage remote exploit. The amount of engineering required to pull these stunts off - reliably no less - is truly staggering, and yet attackers continue to innovate and develop new techniques and even more complex exploits. And you overestimate the competency of the defenders: sysadmin? What small business has a dedicated sysadmin? It’s not like we are talking about running a global hosted chat service like slack here. (Oh wait...) BTW Persistence for the key in memory is trivial: create a shared memory segment and store the key there, with a ttl.
- mgleason_3 7y agoBackup is on a separate network. The network is segregated to limit impact if they do get hacked. To keep everyone familiar, Data Recovery processes are practiced regularly. ...