About 10 years ago I saw a few emails regarding broken video clips. Ignored them as it’s always a local problem (like the volume isn’t plugged in) Eventually i

About 10 years ago I saw a few emails regarding broken video clips. Ignored them as it’s always a local problem (like the volume isn’t plugged in) Eventually it got passed to me. The files were missing. This was on a 16T file store, one of about 10, and investigating showed about 200G was missing. Backup was an rsync job every hour to another server, with a —-delete option every week on a Monday morning. Trawling though the auth logs eventually showed a second line engineer had elevate shimmed to root and managed to delete a bunch of files by running “find -delete” with a mangled name filter. He’d tried to solve a real problem by copying and pasting from google, failed, not realised why it failed, and given up. Several dozen hours of archive material had been gone, forever (There was also the time a different engineer at a different site transferred an archive with ftp. In ascii mode.)