3 ms·

Heh, I'm sure it did. I worked at a few VFX studios, and everyone has deleted large swathes of shit by accident. My favourite was when an rsync script went ro

Heh, I'm sure it did. I worked at a few VFX studios, and everyone has deleted large swathes of shit by accident. My favourite was when an rsync script went rogue and started deleted the entire /job directory in reverse alphabet order. Mama-mia[1] was totally wiped out, as was Wanted (that was missing some backups, so some poor fucker had to go round fishing assets out of /tmp, from around 2000 machines.) From what I recall (this was ~2008) There was some confusion as to what was doing the deleting. Because we had at the time a large(100 or 300tb[2]) lustre file system, it didn't really give you that many clues. They had to wait till it went on a plain old NFS box before they could figure out what was causing it. Another time honoured classic is matte painters on OSX boxes accidentally dragging whole film directories into random places. [1]some films have code names, hence why this was first [2]That lustre was big, physically and IO, it could sustain something like 2-5 gigabytes a second, It had at least 12 racks of disks. Now a 4u disk shelf and one server can do ~2gigabyes sustained

KaiserPro 10y ago
In the most recent VFX company I worked at, with some of the same guys, the backup sys was fucking ace. Firstly rm was aliased to a script that just moved stuff, not deleted it. Second, there were very large nearlines that took hourly snapshots. Finally, lots and lots of tape for archive.
mprovost 10y ago
We lost a good chunk of Tintin (I think) when someone tried to use the OSX migration assistant to upgrade a Macbook that had some production volumes NFS mounted. It was trying in vain to copy several PB of data (I am convinced that nobody at Apple has ever seen or heard of NFS), and because it was so slow the user hit cancelled and it somehow tried to undo the copy and started deleting all the files on the NFS servers. There was another incident where there was a script that ran out of cron to cleanup old files in /tmp, and someone NFS mounted a production volume into /tmp... Eventually we put tarpit directories at the top of each filesystem (a directory with 1000 subdirectories each with 1000 subdirectories, several layers deep) to try and catch deletes like the one you saw, then we would alert if any directories in there were deleted so we could frantically try and figure out which workstation was doing the damage.
- cookiecaper 10y ago
  I had a client with a Linux server who wanted to automount the share on their OS X workstations. I cannot believe the hoops I had to jump through to make something as simple as NFS work. Every iteration of OS X seems to make traditional *nix utilities less and less compatible and remove valuable tools for no reason other than obstinance.

KaiserPro 10y ago

In the most recent VFX company I worked at, with some of the same guys, the backup sys was fucking ace. Firstly rm was aliased to a script that just moved stuff, not deleted it. Second, there were very large nearlines that took hourly snapshots. Finally, lots and lots of tape for archive.

mprovost 10y ago

We lost a good chunk of Tintin (I think) when someone tried to use the OSX migration assistant to upgrade a Macbook that had some production volumes NFS mounted. It was trying in vain to copy several PB of data (I am convinced that nobody at Apple has ever seen or heard of NFS), and because it was so slow the user hit cancelled and it somehow tried to undo the copy and started deleting all the files on the NFS servers. There was another incident where there was a script that ran out of cron to cleanup old files in /tmp, and someone NFS mounted a production volume into /tmp... Eventually we put tarpit directories at the top of each filesystem (a directory with 1000 subdirectories each with 1000 subdirectories, several layers deep) to try and catch deletes like the one you saw, then we would alert if any directories in there were deleted so we could frantically try and figure out which workstation was doing the damage.

cookiecaper 10y ago
I had a client with a Linux server who wanted to automount the share on their OS X workstations. I cannot believe the hoops I had to jump through to make something as simple as NFS work. Every iteration of OS X seems to make traditional *nix utilities less and less compatible and remove valuable tools for no reason other than obstinance.