3 ms·
I feel like these are self-contradictory unless you're really going to go through the effort of periodically restoring each of your offline tapes to a server (w
by yellowapple 8y ago
I feel like these are self-contradictory unless you're really going to go through the effort of periodically restoring each of your offline tapes to a server (which I'm sure some organizations end up doing, to be fair, but that doesn't make it any less of a pain in the ass).
- AznHisoka 8y agoI also hear people say that all the time. Usually they're managing measly small 1-100 GB databases, not multiple ones > 1 TB in size. Restoring them in a regular basis is easy in that case. But I'd love to hear how people practice restoring all their backups on a regular basis when it requires having an enormous amount of space. Practically, I just restore a few backups every month, and if it works, I just assume the others work as well (which I know is a false assumption, but what are you going to do?)
- Intermernet 8y agoMy DR procedure was to automatically restore the mail server, DB server and file server once a month with automated fail back to the live servers if they failed. These would happen outside heavy operation hours, and if successful, the restored servers would become the live servers in most instances. These were done using scripted vmware instances, mostly restoring from StorageCraft Shadow Protect backups. File server was the most resource intensive and the most prone to complaints about downtime, but usually the trade off was fine. DB server was usually fine due to the out of hours restore, and the ability to master - slave - replicate - handover. Mail server was usually transparent as you could set up smtp transfer rules for incoming, and outgoing didn't really care. The trickier ones were things like IBM I Series servers that usually require periodic full system saves, restores and tests. These couldn't be done automatically without excessive licensing fees and much manual baby sitting. If you get to the point where you have automatic scheduled DR, you can be pretty confident your backups are working. It the automated DR fails, you roll back, learn whatever lesson the failure taught you, and change the system so it doesn't happen again. Summary: schedule DR, not backups.
- Dylan16807 8y agoI'm confused. Why does the number of restores affect how much space you need?
- AznHisoka 8y agoYou're right, we could restore each, one by one. However, we have around 30 different databases, each of which take more than a day to restore. We backup databases every X days, upload them to S3, and delete the previous ones, keeping only the last 3 or so. So if it takes us 30 days to restore all the databases, those backups might already have been gone in S3 as part of the cleanup operation anyway. It just gets complicated, and non-trivial when you have multiple databases, each TB's large.
- cheald 8y agoIf you already have the DBs in S3, it seems obvious that you could bring up EC2 instance(s) with gigantic EBS drives attached, restore a DB to it, and run verification measures on it. Parallelize as your budget allows.
- AstralStorm 8y agoEC2 is not an airgapped machine. That's known as cloud backup not offline backup. The solution is to have a tape or disk library and have the other machine take over for restore. If you have TBs of data you can afford a library and another PC to run the recovery test.
- Dylan16807 8y ago> EC2 is not an airgapped machine. That's known as cloud backup not offline backup. Nobody said it was. This subthread is about the importance of testing backups in general. > If you have TBs of data you can afford a library and another PC to run the recovery test. They already explained how that's not enough. Yes, they can add more, but you're missing some info there.
- ifoundthetao 8y agoYou need to test your backups. It's a pain, but it'll save you when you need it.
- gav 8y agoThere has to be something better than tape, I know I'm just a single data point, but I've had _terrible_ luck with tapes over the years. I've tried various options over the years but I haven't found a simple OTS non-tape offline backup solution. There's a lot of potential here for a solution.
- theandrewbailey 8y agoSince you're talking about tapes, I'm guessing there's more data than can be put on an external hard drive?
- gav 8y agoLTO-8 tapes can store 12TB (uncompressed) for about $300/ea, which is about 30% less than a disk, with supposedly a 30-year life. The drives start at around $5000.
- cr0sh 8y ago30-year life when the tapes are properly stored in certain environmental conditions. Usually those conditions aren't met properly.
- yellowapple 8y agoI only mentioned tapes specifically because they're probably the most common choice for offline backups (I say "probably" because I don't know the relative numbers between tapes and, say, optical media). You're definitely right, though; there's gotta be a better way. SSDs look promising, though not necessarily for long-term storage (for that, I suspect it'd be hard to beat optical, aside perhaps from printing out a bunch of QR codes or something).
- AstralStorm 8y agoSSD are considered a fail for backup since they do have to be powered on to do maintenance to survive years of storage. However, since you're scheduling recovery tests, might as well run a SMART offline full on the drive to be sure then. More often than a year is good.
- Dylan16807 8y agoRegular tests are fine. Restoring every single backup is unwarranted paranoia. It doesn't prove that your system is failproof, because nothing is.
- arethuza 8y agoIt may seem like a pain in the ass - but wait until you try restoring a system from a backup tape and the first tape doesn't work, neither does the second, or the third ..... Learned that lesson early in my career - the fourth and final tape did restore. NB The backups weren't my responsibility, but restoring the payroll database my shell scripting error had nuked most definitely was.
- yellowapple 8y agoThat's the thing about ass-pains, though: humans are prone to simply not do them. That's why the trick to getting humans to Do The Right Thing™ is to make it as easy as possible to do so.
- AstralStorm 8y agoYou try to restore the backup to an airgapped machine separately set out for that. This tests the worst case: server burns down, new machine is in.