A few years ago I worked on a research project at a university where we generated quite a bit of data, and we would occasionally (every few months) run an archi

A few years ago I worked on a research project at a university where we generated quite a bit of data, and we would occasionally (every few months) run an archiving script which would copy data from one MySQL table to another and then truncate the source table. I once ran this without remembering that the columns in the source table had been changed a few weeks previously and the destination table and script had not been updated accordingly. Result: The insert into the new table failed but the old one was truncated anyway. This shouldn't have been too much of a problem because a full dump of the MySQL database was made every night, copied offsite with scp, and kept for a week. But, when the time came to restore the affected table from one of these dumps I found that every dump for the last week had timed out silently and was incomplete. The result was the best part of a week trying to reconstruct the missing parts of the table from what could be found in the dumps and to restore it successfully. I got most of it back, but we couldn't do much work for that period.