3 ms·
Let's be clear - nobody cares about backups. Any seasoned SysAdmin (That is, one who has been axed, or been on a team where someone was axed for not having a p
by ghshephard 15y ago
Let's be clear - nobody cares about backups. Any seasoned SysAdmin (That is, one who has been axed, or been on a team where someone was axed for not having a proper recovery process in place) will tell you that restores (and the testing of said restores) is what is essential.
Weekly/monthly testing of restores is critical - preferably in an automated fashion.
People would be amazed to know how many large enterprises have invested 10s of millions of dollars in large tape array/robots, which they religiously back up petabytes of data to, without actually knowing whether they'll be able to restore properly should the day come that they need to.
- harshreality 15y agoAs an example of why this is so important, consider a classic case: Amazon's May 1997 database snafu, documented by Jeremiah Wilton at http://www.bluegecko.net/news-events/new-whitepaper-the-dba-disaster-diary-double-failure/ http://www.bluegecko.net/news-events/new-whitepaper-the-dba-... Frequent automated restoration and testing of database backups would have avoided the downtime. Amazon would have noticed the Oracle bug soon after the extents change, and would have had a chance to identify and work around it before taking down the production database for routine maintenance. Furthermore, by using their backup DEC storage array for the routine backup restoration tests, they would have identified their new DEC storage array as faulty before they put it into production use.
- andrewflnr 15y agoOkay, total sysadmin noob here: how do you usually test restoration? If I was to test HDD image restoration on my laptop, for example, and it didn't work, I would be doubly up a creek, because not only is my disk hosed, but I've just proven I can't restore it. On our home server it's not quite as bad, but there's only so much you can do with dry runs. I suppose if it's just the files you're interested in, it's not as hard, but much more of a pain if you mess up your OS.
- larrys 15y agoA hard drive with multiple partitions or another boot drive would be a good start. A good tool for the Mac by the way (if you use OSX) is "Super Duper" which will clone your hard drive. Then you can simply pick the cloned drive as your boot drive and if it boots you will know you have a good backup. That is how I test that my backup works. http://www.shirt-pocket.com/SuperDuper/SuperDuperDescription.html http://www.shirt-pocket.com/SuperDuper/SuperDuperDescription... When I was recently traveling I took several pocket hard drives with clones of my laptop drive so if the drive failed (or the laptop failed) I could hook the hard drive up to another laptop (one rented or purchased) and boot from that.
- andrewflnr 15y agoNot using OSX, but I do have an external drive that I might be able to make bootable backups on if I can find the right tools. Sounds like a useful trick regardless.
- harshreality 15y agoWithout duplicate hardware (at least a duplicate hdd), obviously you can't test restoring a bare metal OS environment without some risk. For business purposes where maximum performance is not so critical, running necessary apps in a VM avoids the problem you describe.
- larrys 15y ago"Without duplicate hardware (at least a duplicate hdd)" You can do this with multiple partitions on the same physical drive assuming you have the space available.
- gojomo 15y agoIndeed, and this goes for home backups too. You never know something is there unless you test recovering it. For example: Apple's Time Machine seems like a slick, safe way to backup your entire hard drive. And, you can use it to transfer the entire old drive to a new drive. But, if there is an I/O error reading any of the files on backup, it will simply skip that file, and the only indication of a problem is in the /var/log/system.log. It can skip the file for months... eventually reaching the point where any older backup where the file was successfully read is aged out. And even on recovery: the file was skipped, so there's no indication it's even missing. Only when an app needs it is there any indication that months of backups, and a supposed full-recovery, had silent failures.
- larrys 15y agoI had a problem with time machine that caused me to stop using it. When I switched (think it was under 10.5) from regular to filevault it stopped doing backups. I happened to pay attention to the time that it took and noticed it seemed extremely quick. I decided (as I mentioned in another reply here) to use Super Duper which makes a clone of your hard drive and is easy to verify (just boot from the cloned drive).
- chetan51 15y agoIs this the case even in the latest versions of the OS and Time Machine? I see an small app opportunity here.
- gojomo 15y agoThrough lion at least. There are a couple 3rd-party apps to address: one's called 'Time Machine Buddy' and another is 'Log Viewer for Time Machine'. Neither is super-slick (or seems to do anything more than grepping /var/log/system.log for backupd info), but they cover the basic need... once you know it's an issue, which may only happen after being bitten.
- larrys 15y ago"that restores (and the testing of said restores) is what is essential" Not sure if it's true or urban legend but I remember reading a story about 15 years ago about tape backups being left on the seat of a car with heated seats. When they finally needed them they had been erased by the electric current in the seats. (That's the rough story that I remember at least).
- ars 15y agoIt's probably not true. The magnetic field generated by heated seats is very very low. The coercivity of magnetic tape is much higher than that, plus the car field is constant, which doesn't work very well to erase thing. The heat maybe, that could do it, magnetic media doesn't like heat. A car warmer would not be enough, but a hot car in the summer might be.
- larrys 15y agoHaving bought an tape eraser at radio shack some time ago I would tend to agree with you. To me though the story has value and sticks out in my mind simply to reinforce unknowns that should cause you to always verify backups. Has much more impact than simply reading "important to test backups". That said the tapes I believe were large reels and the seats were an old Saab or something like that. So who knows the force field that thing gave off.