3 ms·

In the past years, I've grown increasingly concerned about backups. To me, it feels like whatever you back up should be validated, before it's considered "okay"

In the past years, I've grown increasingly concerned about backups. To me, it feels like whatever you back up should be validated, before it's considered "okay". So, if you have a database of some sort, a part of the backup process would be checking that the backup can be used to run an instance of it. If you have images or videos or PDF files, all of those should be validated as well, to make sure that they're not corrupted (e.g. display/playback fails). If you have compression as a part of the backup process, then the compressed archive should also be validated, that it's not corrupt and that the data can also be extracted. Only then would checksums of the backup be actually useful, once you'd know that you don't just have a bunch of mostly useless binary data on hand. Sadly, the tooling just isn't there (e.g. CLI utilities for validating everything) and there are far more file formats than most people want to concern themselves with.

dolmen 4y ago
Well I would say that you must ensure that using the backup will solve the problem before choosing that solution. This implies that you must take time to analyze the source of the problem first.
marginalia_nu 4y ago
Seems like it would be prohibitively time-consuming. For a moderately large data-set (of say a few hundred Tb), by the time the backup of your data is vetted and validated, it's months old and effectively useless.
- ilyt 4y ago
  Yet you still need to do it, even if it is every few months, else you will eventually hit the snag. If you don't have an option to restore the data your backup might as well not exist.
capableweb 4y ago
> In the past years, I've grown increasingly concerned about backups. To me, it feels like whatever you back up should be validated, before it's considered "okay". I've been trying to toot this horn at every place I've consulted/worked at, with the most common response from software engineers being: "We're using the AWS backup system of the database, of course it works, no need to test it" People have way too much trust in systems overall, especially ones they have no inside information about. And people also seems to default to "Of course it works" without testing it, instead of "How would I know it works if I haven't tested it?" or even "I don't know".
nop_slide 4y ago
Related: http://www.taobackup.com/ http://www.taobackup.com/
- aftbit 4y ago
  Thanks! I haven't seen this before, though I have heard these 7 general principles. My personal strategy is broadly: 1. daily encrypted restic backups to my local NAS, running ZFS raidz2 2. weekly rsyncs of the backup directory to an offsite machine 3. yearly full offsites of the entire NAS I test my backups frequently if only because I have some odd persistent bug that eats my zsh history file about once a month. I now have a "recover-latest-history" script that pulls that file back down from the latest restic backup. I have just completed an offsite and exchanged it for last year's, so I can test that now too! Rate my tao :)

dolmen 4y ago

Well I would say that you must ensure that using the backup will solve the problem before choosing that solution. This implies that you must take time to analyze the source of the problem first.

marginalia_nu 4y ago

Seems like it would be prohibitively time-consuming. For a moderately large data-set (of say a few hundred Tb), by the time the backup of your data is vetted and validated, it's months old and effectively useless.

ilyt 4y ago
Yet you still need to do it, even if it is every few months, else you will eventually hit the snag. If you don't have an option to restore the data your backup might as well not exist.

capableweb 4y ago

> In the past years, I've grown increasingly concerned about backups. To me, it feels like whatever you back up should be validated, before it's considered "okay". I've been trying to toot this horn at every place I've consulted/worked at, with the most common response from software engineers being: "We're using the AWS backup system of the database, of course it works, no need to test it" People have way too much trust in systems overall, especially ones they have no inside information about. And people also seems to default to "Of course it works" without testing it, instead of "How would I know it works if I haven't tested it?" or even "I don't know".

nop_slide 4y ago

Related: http://www.taobackup.com/ http://www.taobackup.com/

aftbit 4y ago
Thanks! I haven't seen this before, though I have heard these 7 general principles. My personal strategy is broadly: 1. daily encrypted restic backups to my local NAS, running ZFS raidz2 2. weekly rsyncs of the backup directory to an offsite machine 3. yearly full offsites of the entire NAS I test my backups frequently if only because I have some odd persistent bug that eats my zsh history file about once a month. I now have a "recover-latest-history" script that pulls that file back down from the latest restic backup. I have just completed an offsite and exchanged it for last year's, so I can test that now too! Rate my tao :)