I used to manage a set of tape backup robots (huge square tapes, I want to say 25 glorious GB a piece, and some Sony DAT devices @50-75GB a piece) that backed u

I used to manage a set of tape backup robots (huge square tapes, I want to say 25 glorious GB a piece, and some Sony DAT devices @50-75GB a piece) that backed up sister data centers over a few DS-3s back in the early 2000s. The hardware/servers involved were around a 3/4 million. I only provide that context because I've been as far away from that experience as I can possibly put myself and I can only hope it's gotten a lot better. Prior to me and during the time that I managed backups, the statement "if you don't have backups at all, you don't have backups" was true for the entire backup set probably about once a week[0]. Those miserable robotic devices would crush tapes, the drives would eat tapes or otherwise fail, the backup servers would be over-whelmed (despite being nearly the largest hosts we had), the software would crash (Backup Exec), the server would just be ... skipped? And the tapes had a lifetime that was 1/10th what was advertised (and 1/2 what we had planned on[1]). Our process included a monthly restore test of a few critical pieces of infrastructure and a rotating list of other hosts. I think the one time that we passed might have been worthy of opening a bottle of wine. Most of the time we could get enough data back that the restore wouldn't result in more than a rounding-error financial loss, but for the most part our server team spent a lot of time praying. That system gave me nightmares for years after I left that position. [0] It was so bad that our initial plan was to backup DC A using DC B and vice versa, but we opted to backup both from both data-centers (one of our requirements was off-site backups). [1] I think I read the headline to an article a month ago about why "tape is still king" in backups. I couldn't bring myself to click-in. I assume things have gotten better, but if they're still selling this product the way they did back then, I'd want real-world numbers on MTBF and the like.