4 ms·
It's important to check that not only can you restore the database but you also check that it contains the data you think it contains. Finding out that your ba
by arethuza 11y ago
It's important to check that not only can you restore the database but you also check that it contains the data you think it contains.
Finding out that your backup consists of a perfectly backed up empty database isn't much fun!
- wlll 11y agoOne of the first things I did at my current place was to add a Nagios check "Is the most recent backup size within x bytes of the previous one". It wasn't comprehensive, but was fast to implement and was later replaced with full restores and checks, but nevertheless a few weeks later it caught a perfectly formed tar.gz of a completely empty database due to a bug that would have continued creating empty archives until it was fixed.
- saidajigumi 11y agoThis is a fantastic illustration of testing 101: often it's the dumbest possible checks catch huge errors. Get those in place before overthinking. E.g. a team adjacent to mine years ago had a dev who made a one-character typo in a commit that went to production. Which caused many $MM to incorrectly flow out the door post-haste. The bad transactions were fortunately reversible with some work, but I was floored that there were no automated tests gating these changes. It wasn't a subtle problem. The most basic, boring integration test of "run a set of X transactions, check the expected sum" would have prevented that failure.
- chris_wot 11y agoAll these threads are complete gold mines of years of hard won experience and system administration tips! Fascinating, especially the war stories I'm reading - both hilarious and horrifying in equal measure.
- KB1JWQ 11y agoAnd as Gliffy is discovering to their detriment, exactly how long said restore takes...
- vonklaus 11y agoResonates. I didn't spend the extra $5 to get a USB 3.0 Flash Drive. Currently the read process for the image backup looks like it will take 4 hours to create write media, 2 to actually restore. Good lesson above about te restore being more inportant than backup. Wish I learned that.