4 ms·

And the most important part of the rule which they forgot: Test your backups. I once has to restore a DB from backups just to discover that the backups from 2

And the most important part of the rule which they forgot: Test your backups. I once has to restore a DB from backups just to discover that the backups from 2 days ago were corrupted. Fortunately backup t-3 days worked. And I had to do some binlog mongering to recover the rest of the data. Test your backups people. They WILL fail.

ws66 5y ago
From someone smarter than me: people think they need backups while in fact they need restores…
- I_complete_me 5y ago
  Beautifully succinct.
- randombits0 5y ago
  That’s one of my preso jokes. “Backups are a worthless waste of time! Restores, on the other hand…”
- oxfeed65261 5y ago
  The article slightly annoyed me by saying, “Why one does backups? To have a copy of data at hand if something goes wrong.” Not, I think, the best way to put it, per your comment.
- still_grokking 5y ago
  The canonical link: https://www.joelonsoftware.com/2009/12/14/lets-stop-talking-about-backups/ https://www.joelonsoftware.com/2009/12/14/lets-stop-talking-...
  - bradknowles 5y ago
    In the late 1990s, W. Curtis Preston wrote a book called “Unix Backup and Recovery”. A later version was just named “Backup and Recovery”. Both were published by O’Reilly. He was saying similar things at that time (e.g., “Backups are worthless, Restores are priceless”), and I’m certain he wasn’t the first. But you’d have to ask him who was saying things like that before he did. Disclaimer: Curtis was a co-worker of mine at the time, and I was a technical reviewer of the book in question.
laurent123456 5y ago
How do you go about testing your backups? Like if you have 1TB of data, testing that all of it is valid is not trivial, and I guess that should be done every time you create a new backup set? Shouldn't the backup tool somehow perform this test automatically?
- mytailorisrich 5y ago
  There is no alternative to performing an actual restore periodically in order to verify that backups and the restore procedure do work.
- throw0101a 5y ago
  > testing that all of it is valid is not trivial If it was easy then people wouldn't have to be reminded to do it—it'd just get done. :) One way could be to pre-compute checksums of the files in question and then verify them periodically: * https://aide.github.io https://aide.github.io * https://packages.debian.org/search?keywords=aide https://packages.debian.org/search?keywords=aide It's also why people suggest using ZFS: it has checksums on the live data to make sure bits aren't flipped, but those checksums can then be replicated to remote copies (zfs send/recv) to make sure your copies also coherent. A service like rsync.net allows remote ZFS data sets—that can also be encrypted.
  - bigiain 5y ago
    Being 100% confident you've got a bit for bit copy of what you think you have, doesn't mean you have a restore process that works. Unless you've tested and proven to your self you can bring up a working system from your backups, you've only done the first half of your disaster recover work. For me, that means restoring onto your cold spare hardware (either identical or similar-enough for you), shutting down your system pulling it's drives and replacing them with blank ones and restoring onto that (but be careful you can be sure you don't need that specific hardware/firmware/peripherals - cause in 18 months time when someone backs a truck into your office and load all your electronics into it you might not be able to get an identical system), or restoring a running system onto newly provisioned instances on cloud-provider-of-choice. Prove to yourself at least, that you can have business continuity in a know amount of time once you pull the pin on your DR/restore-from-backups plan.
digitallyfree 5y ago
Not just test the backup, but practice and ensure you are familar with the recovery process as well. Having a clear plan of what you will do if that disk fails/machine is lost/etc. is better than troubleshooting and fumbling with tools while the clock is ticking. A written procedure may be helpful as well (I have one for my Proxmox server for setting up the machine and restoring the VMs from the backup disk).

ws66 5y ago

From someone smarter than me: people think they need backups while in fact they need restores…

I_complete_me 5y ago
Beautifully succinct.
randombits0 5y ago
That’s one of my preso jokes. “Backups are a worthless waste of time! Restores, on the other hand…”
oxfeed65261 5y ago
The article slightly annoyed me by saying, “Why one does backups? To have a copy of data at hand if something goes wrong.” Not, I think, the best way to put it, per your comment.
still_grokking 5y ago
The canonical link: https://www.joelonsoftware.com/2009/12/14/lets-stop-talking-about-backups/ https://www.joelonsoftware.com/2009/12/14/lets-stop-talking-...
- bradknowles 5y ago
  In the late 1990s, W. Curtis Preston wrote a book called “Unix Backup and Recovery”. A later version was just named “Backup and Recovery”. Both were published by O’Reilly. He was saying similar things at that time (e.g., “Backups are worthless, Restores are priceless”), and I’m certain he wasn’t the first. But you’d have to ask him who was saying things like that before he did. Disclaimer: Curtis was a co-worker of mine at the time, and I was a technical reviewer of the book in question.

laurent123456 5y ago

How do you go about testing your backups? Like if you have 1TB of data, testing that all of it is valid is not trivial, and I guess that should be done every time you create a new backup set? Shouldn't the backup tool somehow perform this test automatically?

mytailorisrich 5y ago
There is no alternative to performing an actual restore periodically in order to verify that backups and the restore procedure do work.
throw0101a 5y ago
> testing that all of it is valid is not trivial If it was easy then people wouldn't have to be reminded to do it—it'd just get done. :) One way could be to pre-compute checksums of the files in question and then verify them periodically: * https://aide.github.io https://aide.github.io * https://packages.debian.org/search?keywords=aide https://packages.debian.org/search?keywords=aide It's also why people suggest using ZFS: it has checksums on the live data to make sure bits aren't flipped, but those checksums can then be replicated to remote copies (zfs send/recv) to make sure your copies also coherent. A service like rsync.net allows remote ZFS data sets—that can also be encrypted.
- bigiain 5y ago
  Being 100% confident you've got a bit for bit copy of what you think you have, doesn't mean you have a restore process that works. Unless you've tested and proven to your self you can bring up a working system from your backups, you've only done the first half of your disaster recover work. For me, that means restoring onto your cold spare hardware (either identical or similar-enough for you), shutting down your system pulling it's drives and replacing them with blank ones and restoring onto that (but be careful you can be sure you don't need that specific hardware/firmware/peripherals - cause in 18 months time when someone backs a truck into your office and load all your electronics into it you might not be able to get an identical system), or restoring a running system onto newly provisioned instances on cloud-provider-of-choice. Prove to yourself at least, that you can have business continuity in a know amount of time once you pull the pin on your DR/restore-from-backups plan.

digitallyfree 5y ago

Not just test the backup, but practice and ensure you are familar with the recovery process as well. Having a clear plan of what you will do if that disk fails/machine is lost/etc. is better than troubleshooting and fumbling with tools while the clock is ticking. A written procedure may be helpful as well (I have one for my Proxmox server for setting up the machine and restoring the VMs from the backup disk).