22 ms·

Did Pixar accidentally delete Toy Story 2 during production? (2012)

515 points by chenster 10y ago

1denis 10y ago
video link in the article is dead
- deleted 10y ago
  [deleted]
hybridtupel 10y ago
The video link is broken. Here is (what seems to be) the same video: https://www.youtube.com/watch?v=8dhp_20j0Ys https://www.youtube.com/watch?v=8dhp_20j0Ys
franze 10y ago
here is a good writeup http://thenextweb.com/media/2012/05/21/how-pixars-toy-story-2-was-deleted-twice-once-by-technology-and-again-for-its-own-good/?amp=1 http://thenextweb.com/media/2012/05/21/how-pixars-toy-story-...
dvdhnt 10y ago
From the next web write up - > The command that had been run was most likely ‘rm -r -f *’, which—roughly speaking—commands the system to begin removing every file below the current directory. This is commonly used to clear out a subset of unwanted files. Unfortunately, someone on the system had run the command at the root level of the Toy Story 2 project and the system was recursively tracking down through the file structure and deleting its way out like a worm eating its way out from the core of an apple.
- wmil 10y ago
  It's more likely they wanted to remove all hidden directories from their home directory, and ran `rm -rf .*`. Under some shells (eg bash) that will expand to include `..` and `.`.
  - ChristianBundy 10y ago
    Why is that more likely?
  - reacweb 10y ago
    The last time it happened to me, it was caused by a space. Instead of rm -rf toto٭, I had rm -rf toto ٭. Hopefully, it was not at the root directory and we have frequent snapshots.
  - aptwebapps 10y ago
    I have totally never done that. If I had done that, I might have been saved by a lack of permissions. Like if I was on a mounted external drive, so not in my home directory and it didn't get too far. Edit: What would have been much more worse, if I had done it, would have been rm -rf ~ /foo instead of rm -rf ~/foo
    - majewsky 10y ago
      Such a bug was in an Nvidia driver install script a few years ago: rm -rf /usr /lib/modules/something/something
      - yomly 10y ago
        That's mortifying!
      - czinck 10y ago
        Bumblebee specifically, https://github.com/MrMEEE/bumblebee-Old-and-abbandoned/commit/a047be85247755cdbe0acce6f1dafc8beb84f2ac https://github.com/MrMEEE/bumblebee-Old-and-abbandoned/commi... is the fix. Pretty sure Steam had the same issue early in their linux release, probably lots of other programs too.
        cookiecaper 10y ago
        Yes, I specifically remember a lot of hub bub about Steam deleting entire home directories if a path was input incorrectly. Glad that never bit me.
- cobbzilla 10y ago
  obligatory call for everyone to use safe-rm [1]. it's one thing to shoot yourself in the foot, but without safe-rm you (or someone else less cautious) will eventually fire the accidental head-shot. it's happened to me a couple times; but never again since I started using safe-rm everywhere. [1] http://serverfault.com/questions/337082/how-do-i-prevent-accidental-rm-rf http://serverfault.com/questions/337082/how-do-i-prevent-acc...
  - hh2222 10y ago
    I'll add another obligatory, make sure you have a working backup system.
    - esquevin 10y ago
      And that you already tried to restore from it
- knodi123 10y ago
  As a linux neophyte, I once struggled to remember whether the trailing slash on a directory was important. So I typed "rm -rf", and pasted the directory name "schoolwork/project1 " (with a trailing space), but then I waffled and decided to add a trailing slash. So I changed it to "rm -rf schoolwork/project1 /". That's my theory as to what they did.
  - Symbiote 10y ago
    rm -Rf test / rm: it is dangerous to operate recursively on ‘/’ rm: use --no-preserve-root to override this failsafe GNU rm, but I don't know when it was introduced.
    - r3bl 10y ago
      Quick googling around, and it seems like this behavior (in Ubuntu) changed somewhere near the end of 2008: https://bugs.launchpad.net/ubuntu/+source/coreutils/+bug/174283/comments/10 https://bugs.launchpad.net/ubuntu/+source/coreutils/+bug/174... tl;dr That's when coreutils Ubuntu package switched from "--no-preserve-root" to "--preserve-root" by default.
    - blitzkrieg3 10y ago
      This is relatively recent, and caused quite a stir when it was introduced. Certainly not around during Toy Story 2. edit: More info: https://lwn.net/Articles/327141/ https://lwn.net/Articles/327141/
    - coldpie 10y ago
      One feature I've always wanted is to make a directory non-deletable without otherwise changing the directory's functionality. So if I do "rm -rf ~/work" I don't lose anything, but I can still do "touch ~/work/whatever". As far as I know, this is impossible.
      - slau 10y ago
        You don't actually need write access to a file/directory to delete it. You only need write access to the parent (removing or adding a file modifies the container, not the contained). Remove write access to the parent directory, while keeping traversal and read rights, and you'll be sorted.
abe_duarte 10y ago
This event is explained more in depth in the book Creativity Inc by Ed Catmull. It's a pretty good story.
- creaghpatr 10y ago
  love that book
smcl 10y ago
"And then, some months later, Pixar rewrote the film from almost the ground up, and we made ToyStory2 again" So that effort to recreate it (not to mention produce it in the first place) was pretty much all for naught? That must have been soul destroying
- dijit 10y ago
  I disagree wholeheartedly, they had a rare chance to rebuild using their acquired knowledge with none of the debt or cruft. "We have to keep this scene even though it's not quite perfect because otherwise it's a waste of money". Maybe this is a bad example actually, movie industry is something you launch and market and leave. But the best architectures I've seen have been demolished, destroyed and rebuilt from the ground up for their purpose. Same with code.
  - integricho 10y ago
    And I'm also pretty sure they reworked their backup and validation tools as well.
  - jnbiche 10y ago
    > But the best architectures I've seen have been demolished, destroyed and rebuilt from the ground up for their purpose. This can also be a destructive siren call: https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/ https://www.joelonsoftware.com/2000/04/06/things-you-should-... But I think it's not absolute. Sometimes rewrites are imperative.
    - esquevin 10y ago
      In the case of toy story 2, I think the analogy would be that they ditched the product they were working on to create a different one. Thus making joel's point non relevant there
      - olavk 10y ago
        It wasn't a new product, they kept a lot of the assets (notably the characters and the universe) but rewrote the script extensively and had to scrap a lot of animation. I don't think an analogy to software rewrite works very well, the domains are so different. For example Joel makes the point that code have a lot of embedded information which you have to rediscover if you rewrite from scratch. This introduces a huge risk and uncertainty. If you rewrite a movie script halfway during production you have to scrap a lot of work but you don't really loose knowledge, so the risk is more manageable.
leecarraher 10y ago
rm just unlinks the files at the inode level, seems like a disk forensics utility like the imagerec suite could have restored alot of the 'lost' data. In fact i've done it on source code after learning that the default behavior of untar was to overwrite all of your current directory structure. since it was text i didnt need anything fancy like imagerec, instead i just piped the raw disk to grep, and looked for parts i knew were in files i needed, then output them and the surrounding data to an external hard drive.
- colanderman 10y ago
  Back then yes. These days with SSDs, the OS will issue trim commands to the disk, zeroing the blocks from the OS point of view, and on SSDs with "secure delete", from a forensics point of view as well.
  - drakenot 10y ago
    That is very interesting, I didn't know there were SSDs with "secure delete." I remember that Apple removed the "Secure Empty Trash" feature in OS X 10.11 because they didn't feel like they could guarantee secure deletion with their new fast SSDs present in most of their computers.
    - throwaway7767 10y ago
      "secure delete" (actually "secure erase") is a term from the ATA standard, it's supported by practically all ATA and later SATA devices for a very long time. The idea was you'd tell the disk to erase everything, it would do the actual deletion in the background but would not allow reading the old data. This was also the way to reset passworded harddrives, you could send a secure erase command without authentication, the drive would wipe itself and once done, the password is gone. Once SSDs arrived on the scene, they were limited by the interface as ATA didn't specify a way to mark sectors as unused. People found that their performance would degrade with use as all the sectors became utilized. But a secure erase command would mark all sectors as erased, so the drive would work like new. Later on, ATA got the TRIM (and later queued versions of TRIM) so the OS can mark specific sectors as erased. But the result is that a lot of people confuse flash sector erasing with secure erase.
woliveirajr 10y ago
The biggest difference, I think, was leaving the hunting for a head for a second moment, or even not doing it at all. Commitment would be very different if people were being asked to help while some heads were rolling. Because you're a real team when everybody is going in the same direction. Any call on "people, work hard do recover while we're after the moron who deleted everything" wouldn't have done it. You just commit to something when you know that you won't be under the fire if you do something wrong without knowing it.
- mr_luc 10y ago
  I remember this from the Field Guide to Understanding Human Error. Making recovering from human error a well-understood process is important, and as you point out, that process will work best if people aren't distracted by butt-covering.
  - ChristianGeek 10y ago
    Link for those interested: https://www.amazon.com/dp/1472439058 https://www.amazon.com/dp/1472439058
- cookiecaper 10y ago
  Ed Catmull discusses this incident thoroughly in Creativity Inc.. He believed seeking retribution for this incident would've been counterproductive and discouraged Pixar's overall ethos as a safe place to experiment and make mistakes. It is this ethos and culture of vociferous, thorough experimentation and casting everyone's performance in the light of "What can we learn from that?" rather than "What ROI did we get from the last 3 months?" that Catmull credits for Pixar's success (paraphrasing here, but I believe this is an accurate summary). Since Catmull has an engineering background (his PhD involved the invention of the Z-buffer, and he was doing computer graphics before anyone knew anything about it), he understands that mistakes and failed projects, when combined with an forthright and collaborative feedback loop, are not problematic detours, but rather necessary mile markers on the path to real innovation. We'd be so much further ahead if we put more men like Catmull in charge of things. The biggest problem with reading Creativity Inc. is that it will rekindle the hope that there may be a sane workplace out there somewhere, when practically speaking, we know that few of us will ever find employment in one. It gave me a number of disquieting feelings as I read that the attributes of a workplace that all good engineers crave actually can and sometimes do exist out there. I had convinced myself that these things were myths, so now I'm sad that my boss isn't Ed Catmull. That said, I do believe some evaluation and/or discipline would've been appropriate in this case, not for the person who accidentally executed a command in the wrong directory, but for the people who were supposed to be maintaining backups and data integrity. Assuming that your primary job duties involve data integrity and system uptime, having non-functional backups of truly critical data stretches beyond the scope of "mistakes" and into the scope of incompetence. It is, I'm sure, very possible that no one was really assigned this task at Pixar and that it would therefore by improper to punish anyone in particular for the failure to execute it, but I do believe there is a limit between mistakes en route to innovation and negligence. My experience has been that most companies strongly take one tack or the other: they either let heads roll for minor infractions (and thus never allow good people to get established and comfortable), or they never fire anyone and let the dead weight and political fiefdoms gum up the works until the gears stop altogether. There needs to be a balance, and that's a very hard thing to find out there.
yawz 10y ago
Funny that this comes up a few weeks after I finished Ed Catmull's "Creativity Inc." If you want a little more detail about this (and other Pixar related things and Steve Jobs) read the book. It is a really good one.
ihaveahadron 10y ago
Didn't toy story one almost get deleted or something.
mentos 10y ago
John Cleese's talk on Creativity recently made it to the front page of HN again https://www.youtube.com/watch?v=9EMj_CFPHYc https://www.youtube.com/watch?v=9EMj_CFPHYc and if you haven't watched it I highly recommend it. I believe it was in this talk that he says the best work he ever did was when he scrapped and started over. Which from practice I think we can all admit that while its the hardest to do, it is always for the best.
- bluejekyll 10y ago
  > ... it is always for the best. Not necessarily. People often underestimate (in engineering fields) how much work it will take to rebuild something. In software there is a high degree of creativity which can have large downstream effects. You need to architect your system in such a way to make it possible to replace components when needed, this is where strong separation of concerns is important. One thing that I've seen happen time and again is an organization bifurcating itself, so that there is one team working on the new cool replacement, and the other working on the old dead thing that everyone hates. Needless to say this creates anymosity and serverely limits an organizations ability to respond to customer demands. Starting over should be taken very seriously.
KaiserPro 10y ago
Heh, I'm sure it did. I worked at a few VFX studios, and everyone has deleted large swathes of shit by accident. My favourite was when an rsync script went rogue and started deleted the entire /job directory in reverse alphabet order. Mama-mia[1] was totally wiped out, as was Wanted (that was missing some backups, so some poor fucker had to go round fishing assets out of /tmp, from around 2000 machines.) From what I recall (this was ~2008) There was some confusion as to what was doing the deleting. Because we had at the time a large(100 or 300tb[2]) lustre file system, it didn't really give you that many clues. They had to wait till it went on a plain old NFS box before they could figure out what was causing it. Another time honoured classic is matte painters on OSX boxes accidentally dragging whole film directories into random places. [1]some films have code names, hence why this was first [2]That lustre was big, physically and IO, it could sustain something like 2-5 gigabytes a second, It had at least 12 racks of disks. Now a 4u disk shelf and one server can do ~2gigabyes sustained
- KaiserPro 10y ago
  In the most recent VFX company I worked at, with some of the same guys, the backup sys was fucking ace. Firstly rm was aliased to a script that just moved stuff, not deleted it. Second, there were very large nearlines that took hourly snapshots. Finally, lots and lots of tape for archive.
- mprovost 10y ago
  We lost a good chunk of Tintin (I think) when someone tried to use the OSX migration assistant to upgrade a Macbook that had some production volumes NFS mounted. It was trying in vain to copy several PB of data (I am convinced that nobody at Apple has ever seen or heard of NFS), and because it was so slow the user hit cancelled and it somehow tried to undo the copy and started deleting all the files on the NFS servers. There was another incident where there was a script that ran out of cron to cleanup old files in /tmp, and someone NFS mounted a production volume into /tmp... Eventually we put tarpit directories at the top of each filesystem (a directory with 1000 subdirectories each with 1000 subdirectories, several layers deep) to try and catch deletes like the one you saw, then we would alert if any directories in there were deleted so we could frantically try and figure out which workstation was doing the damage.
rootbear 10y ago
I was at Pixar when this happened, but I didn't hear all of the gory details, as I was in the Tools group, not Production. My memory of a conversation I had with the main System Administrator as to why the backup was not complete was that they were using a 32-bit version of tar and some of the filesystems being backed up were larger than 2GB. The script doing the backup did not catch the error. That may seems sloppy, but this sort of thing happens in the Real World all the time. At the risk of spilling secrets, I'll tell a story about the animation system, which I worked on (in the 1996-97 time frame). The Pixar animation system at the time was written in K&R C and one of my tasks was to migrate it to ANSI C. As I did that I learned that there were aspects of this code that felt like a school assignment that had escaped from the lab. While searching for a bug, I noticed that the write() call that saved the animation data for a shot wasn't checked for errors. This seemed like a bad idea, since at the time the animation workstations were SGI systems with relatively small SCSI disks that could fill up easily. When this happened, the animation system usually would crash and work would be lost. So, I added an error check, and also code that would save the animation data to an NFS volume if the write to the local disk failed. Finally, it printed a message assuring the animator that her files were safe and it emailed a support address so they could come help. The animators loved it! I had left Pixar by the time the big crunch came in 1999 to remake TS2 in just 9 months, so I didn't see that madness first hand. But I'd like to think that TS2 is just a little, tiny bit prettier thanks to my emergency backup code that kept the animators and lighting TDs from having to redo shots they barely had time to do right the first time. The point is that one would like to think that a place like Pixar is a model of Software Engineering Excellence, but the truth is more complex. Under the pressures of Production deadlines, sometime you just have to get it to work and hope you can clean it up later. I see the same things at NASA, where, for the most part, only Flight Software gets the full on Software Engineering treatment.
- satysin 10y ago
  Cool story :) You are bang on point when it comes to software engineering at what are thought to be "top tier" development houses. In the ideal world sure they will build the very best software but the real world has [unrealistic] deadlines and when you have deadlines it means corners get cut. Not always but very often. This leads to the whole "does it do exactly what is required?" and if it does then you are moved onto the next thing often with the "promise" that you will be able to go back and "fix things" at a later date. Of course we all know that promise is never kept.
colomon 10y ago
This brings up a great practical question. What's the state of the art of this sort of thing for more modest but still modern data storage requirements? Context: For the last five years, my backup system has been to have Time Machine do hourly backups on my MBP (main development machine, just shy of 1TB data), with key spots on my Linux server (3TB data at the moment) backed up daily to my in-laws' house using cron and rsync, and spot directories on the MBP backed up there as well. But the hard drive on the Time Capsule I've used seems to have gotten unreliable, and the external USB drive I bought to replace it has not worked reliably for more than a day or two at a time. And even when it was all working properly, I was never really verifying my backups. Do people have suggestions for secure, reliable, verifiable, easy backup systems capable of handling 4+ TB of data? I don't mind if it takes work or money to set it up; the important thing is once it's working I can mostly forget about it.
- rossng 10y ago
  For an offsite backup, Backblaze is fantastic. Unlimited storage for $5/month and the client works perfectly. It's not highly-redundant or anything, so use it in addition to a local backup. CrashPlan is the next-best option if you need Linux support, but the client isn't as good.
  - buzzybee 10y ago
    Seconding Backblaze for set-and-forget. It's a huge confidence boost for data that is a little bit too large and low-value to handle with more care.
    - colomon 10y ago
      I'm trying Backblaze now, hoping my first backup will be done in a week or so...
- kalleboo 10y ago
  I replaced my Time Capsule with a Synology NAS. Backups are still handled by Time Machine on the Mac (so they're still mostly unobtrusive and idiot-proof), but now they're stored on a mirrored RAID of cheap 3.5" disks. The Synology box is basically just an ARM Linux machine, SSH/root is not locked out if you want it, so if you want to get fancy with off-site backups, you can set up rsync or whatever you want on it. They even ship with some GUIs for mirroring to Dropbox, S3, rsync, another remote Synology NAS, etc.
jrochkind1 10y ago
Oh man, this is the best punch line: > And then, some months later, Pixar rewrote the film from almost the ground up, and we made ToyStory2 again. That rewritten film was the one you saw in theatres and that you can watch now on BluRay. At first I was feeling how it would feel to lose all that work, so frustrating! But then even if you hadn't, it turns out management was gonna throw it all away anyway!
eva1984 10y ago
Good luck. With cloud/distributed computing, I am not sure they can even do that now!
cm2187 10y ago
A good way to fuckup on windows/C#: I Was iterating through network folders to delete (which all start with "\\servername"), except that I had a bug and instead was iterating through the characters of the first network folder path. And that's how I discovered that in windows, "\" means root of the current active drive. And that's also why I value my automatic backup to a NAS twice a day.

515 points by chenster 10y ago

1denis 10y ago

video link in the article is dead

deleted 10y ago
[deleted]

hybridtupel 10y ago

The video link is broken. Here is (what seems to be) the same video: https://www.youtube.com/watch?v=8dhp_20j0Ys https://www.youtube.com/watch?v=8dhp_20j0Ys

franze 10y ago

here is a good writeup http://thenextweb.com/media/2012/05/21/how-pixars-toy-story-2-was-deleted-twice-once-by-technology-and-again-for-its-own-good/?amp=1 http://thenextweb.com/media/2012/05/21/how-pixars-toy-story-...

dvdhnt 10y ago

From the next web write up - > The command that had been run was most likely ‘rm -r -f *’, which—roughly speaking—commands the system to begin removing every file below the current directory. This is commonly used to clear out a subset of unwanted files. Unfortunately, someone on the system had run the command at the root level of the Toy Story 2 project and the system was recursively tracking down through the file structure and deleting its way out like a worm eating its way out from the core of an apple.

wmil 10y ago
It's more likely they wanted to remove all hidden directories from their home directory, and ran `rm -rf .*`. Under some shells (eg bash) that will expand to include `..` and `.`.
- ChristianBundy 10y ago
  Why is that more likely?
- reacweb 10y ago
  The last time it happened to me, it was caused by a space. Instead of rm -rf toto٭, I had rm -rf toto ٭. Hopefully, it was not at the root directory and we have frequent snapshots.
- aptwebapps 10y ago
  I have totally never done that. If I had done that, I might have been saved by a lack of permissions. Like if I was on a mounted external drive, so not in my home directory and it didn't get too far. Edit: What would have been much more worse, if I had done it, would have been rm -rf ~ /foo instead of rm -rf ~/foo
  - majewsky 10y ago
    Such a bug was in an Nvidia driver install script a few years ago: rm -rf /usr /lib/modules/something/something
    - yomly 10y ago
      That's mortifying!
    - czinck 10y ago
      Bumblebee specifically, https://github.com/MrMEEE/bumblebee-Old-and-abbandoned/commit/a047be85247755cdbe0acce6f1dafc8beb84f2ac https://github.com/MrMEEE/bumblebee-Old-and-abbandoned/commi... is the fix. Pretty sure Steam had the same issue early in their linux release, probably lots of other programs too.
      - cookiecaper 10y ago
        Yes, I specifically remember a lot of hub bub about Steam deleting entire home directories if a path was input incorrectly. Glad that never bit me.
cobbzilla 10y ago
obligatory call for everyone to use safe-rm [1]. it's one thing to shoot yourself in the foot, but without safe-rm you (or someone else less cautious) will eventually fire the accidental head-shot. it's happened to me a couple times; but never again since I started using safe-rm everywhere. [1] http://serverfault.com/questions/337082/how-do-i-prevent-accidental-rm-rf http://serverfault.com/questions/337082/how-do-i-prevent-acc...
- hh2222 10y ago
  I'll add another obligatory, make sure you have a working backup system.
  - esquevin 10y ago
    And that you already tried to restore from it
knodi123 10y ago
As a linux neophyte, I once struggled to remember whether the trailing slash on a directory was important. So I typed "rm -rf", and pasted the directory name "schoolwork/project1 " (with a trailing space), but then I waffled and decided to add a trailing slash. So I changed it to "rm -rf schoolwork/project1 /". That's my theory as to what they did.
- Symbiote 10y ago
  rm -Rf test / rm: it is dangerous to operate recursively on ‘/’ rm: use --no-preserve-root to override this failsafe GNU rm, but I don't know when it was introduced.
  - r3bl 10y ago
    Quick googling around, and it seems like this behavior (in Ubuntu) changed somewhere near the end of 2008: https://bugs.launchpad.net/ubuntu/+source/coreutils/+bug/174283/comments/10 https://bugs.launchpad.net/ubuntu/+source/coreutils/+bug/174... tl;dr That's when coreutils Ubuntu package switched from "--no-preserve-root" to "--preserve-root" by default.
  - blitzkrieg3 10y ago
    This is relatively recent, and caused quite a stir when it was introduced. Certainly not around during Toy Story 2. edit: More info: https://lwn.net/Articles/327141/ https://lwn.net/Articles/327141/
  - coldpie 10y ago
    One feature I've always wanted is to make a directory non-deletable without otherwise changing the directory's functionality. So if I do "rm -rf ~/work" I don't lose anything, but I can still do "touch ~/work/whatever". As far as I know, this is impossible.
    - slau 10y ago
      You don't actually need write access to a file/directory to delete it. You only need write access to the parent (removing or adding a file modifies the container, not the contained). Remove write access to the parent directory, while keeping traversal and read rights, and you'll be sorted.

abe_duarte 10y ago

This event is explained more in depth in the book Creativity Inc by Ed Catmull. It's a pretty good story.

creaghpatr 10y ago
love that book

smcl 10y ago

"And then, some months later, Pixar rewrote the film from almost the ground up, and we made ToyStory2 again" So that effort to recreate it (not to mention produce it in the first place) was pretty much all for naught? That must have been soul destroying

dijit 10y ago
I disagree wholeheartedly, they had a rare chance to rebuild using their acquired knowledge with none of the debt or cruft. "We have to keep this scene even though it's not quite perfect because otherwise it's a waste of money". Maybe this is a bad example actually, movie industry is something you launch and market and leave. But the best architectures I've seen have been demolished, destroyed and rebuilt from the ground up for their purpose. Same with code.
- integricho 10y ago
  And I'm also pretty sure they reworked their backup and validation tools as well.
- jnbiche 10y ago
  > But the best architectures I've seen have been demolished, destroyed and rebuilt from the ground up for their purpose. This can also be a destructive siren call: https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/ https://www.joelonsoftware.com/2000/04/06/things-you-should-... But I think it's not absolute. Sometimes rewrites are imperative.
  - esquevin 10y ago
    In the case of toy story 2, I think the analogy would be that they ditched the product they were working on to create a different one. Thus making joel's point non relevant there
    - olavk 10y ago
      It wasn't a new product, they kept a lot of the assets (notably the characters and the universe) but rewrote the script extensively and had to scrap a lot of animation. I don't think an analogy to software rewrite works very well, the domains are so different. For example Joel makes the point that code have a lot of embedded information which you have to rediscover if you rewrite from scratch. This introduces a huge risk and uncertainty. If you rewrite a movie script halfway during production you have to scrap a lot of work but you don't really loose knowledge, so the risk is more manageable.

leecarraher 10y ago

rm just unlinks the files at the inode level, seems like a disk forensics utility like the imagerec suite could have restored alot of the 'lost' data. In fact i've done it on source code after learning that the default behavior of untar was to overwrite all of your current directory structure. since it was text i didnt need anything fancy like imagerec, instead i just piped the raw disk to grep, and looked for parts i knew were in files i needed, then output them and the surrounding data to an external hard drive.

colanderman 10y ago
Back then yes. These days with SSDs, the OS will issue trim commands to the disk, zeroing the blocks from the OS point of view, and on SSDs with "secure delete", from a forensics point of view as well.
- drakenot 10y ago
  That is very interesting, I didn't know there were SSDs with "secure delete." I remember that Apple removed the "Secure Empty Trash" feature in OS X 10.11 because they didn't feel like they could guarantee secure deletion with their new fast SSDs present in most of their computers.
  - throwaway7767 10y ago
    "secure delete" (actually "secure erase") is a term from the ATA standard, it's supported by practically all ATA and later SATA devices for a very long time. The idea was you'd tell the disk to erase everything, it would do the actual deletion in the background but would not allow reading the old data. This was also the way to reset passworded harddrives, you could send a secure erase command without authentication, the drive would wipe itself and once done, the password is gone. Once SSDs arrived on the scene, they were limited by the interface as ATA didn't specify a way to mark sectors as unused. People found that their performance would degrade with use as all the sectors became utilized. But a secure erase command would mark all sectors as erased, so the drive would work like new. Later on, ATA got the TRIM (and later queued versions of TRIM) so the OS can mark specific sectors as erased. But the result is that a lot of people confuse flash sector erasing with secure erase.

woliveirajr 10y ago

The biggest difference, I think, was leaving the hunting for a head for a second moment, or even not doing it at all. Commitment would be very different if people were being asked to help while some heads were rolling. Because you're a real team when everybody is going in the same direction. Any call on "people, work hard do recover while we're after the moron who deleted everything" wouldn't have done it. You just commit to something when you know that you won't be under the fire if you do something wrong without knowing it.

mr_luc 10y ago
I remember this from the Field Guide to Understanding Human Error. Making recovering from human error a well-understood process is important, and as you point out, that process will work best if people aren't distracted by butt-covering.
- ChristianGeek 10y ago
  Link for those interested: https://www.amazon.com/dp/1472439058 https://www.amazon.com/dp/1472439058
cookiecaper 10y ago
Ed Catmull discusses this incident thoroughly in Creativity Inc.. He believed seeking retribution for this incident would've been counterproductive and discouraged Pixar's overall ethos as a safe place to experiment and make mistakes. It is this ethos and culture of vociferous, thorough experimentation and casting everyone's performance in the light of "What can we learn from that?" rather than "What ROI did we get from the last 3 months?" that Catmull credits for Pixar's success (paraphrasing here, but I believe this is an accurate summary). Since Catmull has an engineering background (his PhD involved the invention of the Z-buffer, and he was doing computer graphics before anyone knew anything about it), he understands that mistakes and failed projects, when combined with an forthright and collaborative feedback loop, are not problematic detours, but rather necessary mile markers on the path to real innovation. We'd be so much further ahead if we put more men like Catmull in charge of things. The biggest problem with reading Creativity Inc. is that it will rekindle the hope that there may be a sane workplace out there somewhere, when practically speaking, we know that few of us will ever find employment in one. It gave me a number of disquieting feelings as I read that the attributes of a workplace that all good engineers crave actually can and sometimes do exist out there. I had convinced myself that these things were myths, so now I'm sad that my boss isn't Ed Catmull. That said, I do believe some evaluation and/or discipline would've been appropriate in this case, not for the person who accidentally executed a command in the wrong directory, but for the people who were supposed to be maintaining backups and data integrity. Assuming that your primary job duties involve data integrity and system uptime, having non-functional backups of truly critical data stretches beyond the scope of "mistakes" and into the scope of incompetence. It is, I'm sure, very possible that no one was really assigned this task at Pixar and that it would therefore by improper to punish anyone in particular for the failure to execute it, but I do believe there is a limit between mistakes en route to innovation and negligence. My experience has been that most companies strongly take one tack or the other: they either let heads roll for minor infractions (and thus never allow good people to get established and comfortable), or they never fire anyone and let the dead weight and political fiefdoms gum up the works until the gears stop altogether. There needs to be a balance, and that's a very hard thing to find out there.

yawz 10y ago

Funny that this comes up a few weeks after I finished Ed Catmull's "Creativity Inc." If you want a little more detail about this (and other Pixar related things and Steve Jobs) read the book. It is a really good one.

ihaveahadron 10y ago

Didn't toy story one almost get deleted or something.

mentos 10y ago

John Cleese's talk on Creativity recently made it to the front page of HN again https://www.youtube.com/watch?v=9EMj_CFPHYc https://www.youtube.com/watch?v=9EMj_CFPHYc and if you haven't watched it I highly recommend it. I believe it was in this talk that he says the best work he ever did was when he scrapped and started over. Which from practice I think we can all admit that while its the hardest to do, it is always for the best.

bluejekyll 10y ago
> ... it is always for the best. Not necessarily. People often underestimate (in engineering fields) how much work it will take to rebuild something. In software there is a high degree of creativity which can have large downstream effects. You need to architect your system in such a way to make it possible to replace components when needed, this is where strong separation of concerns is important. One thing that I've seen happen time and again is an organization bifurcating itself, so that there is one team working on the new cool replacement, and the other working on the old dead thing that everyone hates. Needless to say this creates anymosity and serverely limits an organizations ability to respond to customer demands. Starting over should be taken very seriously.

KaiserPro 10y ago

Heh, I'm sure it did. I worked at a few VFX studios, and everyone has deleted large swathes of shit by accident. My favourite was when an rsync script went rogue and started deleted the entire /job directory in reverse alphabet order. Mama-mia[1] was totally wiped out, as was Wanted (that was missing some backups, so some poor fucker had to go round fishing assets out of /tmp, from around 2000 machines.) From what I recall (this was ~2008) There was some confusion as to what was doing the deleting. Because we had at the time a large(100 or 300tb[2]) lustre file system, it didn't really give you that many clues. They had to wait till it went on a plain old NFS box before they could figure out what was causing it. Another time honoured classic is matte painters on OSX boxes accidentally dragging whole film directories into random places. [1]some films have code names, hence why this was first [2]That lustre was big, physically and IO, it could sustain something like 2-5 gigabytes a second, It had at least 12 racks of disks. Now a 4u disk shelf and one server can do ~2gigabyes sustained

KaiserPro 10y ago
In the most recent VFX company I worked at, with some of the same guys, the backup sys was fucking ace. Firstly rm was aliased to a script that just moved stuff, not deleted it. Second, there were very large nearlines that took hourly snapshots. Finally, lots and lots of tape for archive.
mprovost 10y ago
We lost a good chunk of Tintin (I think) when someone tried to use the OSX migration assistant to upgrade a Macbook that had some production volumes NFS mounted. It was trying in vain to copy several PB of data (I am convinced that nobody at Apple has ever seen or heard of NFS), and because it was so slow the user hit cancelled and it somehow tried to undo the copy and started deleting all the files on the NFS servers. There was another incident where there was a script that ran out of cron to cleanup old files in /tmp, and someone NFS mounted a production volume into /tmp... Eventually we put tarpit directories at the top of each filesystem (a directory with 1000 subdirectories each with 1000 subdirectories, several layers deep) to try and catch deletes like the one you saw, then we would alert if any directories in there were deleted so we could frantically try and figure out which workstation was doing the damage.

rootbear 10y ago

I was at Pixar when this happened, but I didn't hear all of the gory details, as I was in the Tools group, not Production. My memory of a conversation I had with the main System Administrator as to why the backup was not complete was that they were using a 32-bit version of tar and some of the filesystems being backed up were larger than 2GB. The script doing the backup did not catch the error. That may seems sloppy, but this sort of thing happens in the Real World all the time. At the risk of spilling secrets, I'll tell a story about the animation system, which I worked on (in the 1996-97 time frame). The Pixar animation system at the time was written in K&R C and one of my tasks was to migrate it to ANSI C. As I did that I learned that there were aspects of this code that felt like a school assignment that had escaped from the lab. While searching for a bug, I noticed that the write() call that saved the animation data for a shot wasn't checked for errors. This seemed like a bad idea, since at the time the animation workstations were SGI systems with relatively small SCSI disks that could fill up easily. When this happened, the animation system usually would crash and work would be lost. So, I added an error check, and also code that would save the animation data to an NFS volume if the write to the local disk failed. Finally, it printed a message assuring the animator that her files were safe and it emailed a support address so they could come help. The animators loved it! I had left Pixar by the time the big crunch came in 1999 to remake TS2 in just 9 months, so I didn't see that madness first hand. But I'd like to think that TS2 is just a little, tiny bit prettier thanks to my emergency backup code that kept the animators and lighting TDs from having to redo shots they barely had time to do right the first time. The point is that one would like to think that a place like Pixar is a model of Software Engineering Excellence, but the truth is more complex. Under the pressures of Production deadlines, sometime you just have to get it to work and hope you can clean it up later. I see the same things at NASA, where, for the most part, only Flight Software gets the full on Software Engineering treatment.

satysin 10y ago
Cool story :) You are bang on point when it comes to software engineering at what are thought to be "top tier" development houses. In the ideal world sure they will build the very best software but the real world has [unrealistic] deadlines and when you have deadlines it means corners get cut. Not always but very often. This leads to the whole "does it do exactly what is required?" and if it does then you are moved onto the next thing often with the "promise" that you will be able to go back and "fix things" at a later date. Of course we all know that promise is never kept.

colomon 10y ago

This brings up a great practical question. What's the state of the art of this sort of thing for more modest but still modern data storage requirements? Context: For the last five years, my backup system has been to have Time Machine do hourly backups on my MBP (main development machine, just shy of 1TB data), with key spots on my Linux server (3TB data at the moment) backed up daily to my in-laws' house using cron and rsync, and spot directories on the MBP backed up there as well. But the hard drive on the Time Capsule I've used seems to have gotten unreliable, and the external USB drive I bought to replace it has not worked reliably for more than a day or two at a time. And even when it was all working properly, I was never really verifying my backups. Do people have suggestions for secure, reliable, verifiable, easy backup systems capable of handling 4+ TB of data? I don't mind if it takes work or money to set it up; the important thing is once it's working I can mostly forget about it.

rossng 10y ago
For an offsite backup, Backblaze is fantastic. Unlimited storage for $5/month and the client works perfectly. It's not highly-redundant or anything, so use it in addition to a local backup. CrashPlan is the next-best option if you need Linux support, but the client isn't as good.
- buzzybee 10y ago
  Seconding Backblaze for set-and-forget. It's a huge confidence boost for data that is a little bit too large and low-value to handle with more care.
  - colomon 10y ago
    I'm trying Backblaze now, hoping my first backup will be done in a week or so...
kalleboo 10y ago
I replaced my Time Capsule with a Synology NAS. Backups are still handled by Time Machine on the Mac (so they're still mostly unobtrusive and idiot-proof), but now they're stored on a mirrored RAID of cheap 3.5" disks. The Synology box is basically just an ARM Linux machine, SSH/root is not locked out if you want it, so if you want to get fancy with off-site backups, you can set up rsync or whatever you want on it. They even ship with some GUIs for mirroring to Dropbox, S3, rsync, another remote Synology NAS, etc.

jrochkind1 10y ago

Oh man, this is the best punch line: > And then, some months later, Pixar rewrote the film from almost the ground up, and we made ToyStory2 again. That rewritten film was the one you saw in theatres and that you can watch now on BluRay. At first I was feeling how it would feel to lose all that work, so frustrating! But then even if you hadn't, it turns out management was gonna throw it all away anyway!

eva1984 10y ago

Good luck. With cloud/distributed computing, I am not sure they can even do that now!

cm2187 10y ago

A good way to fuckup on windows/C#: I Was iterating through network folders to delete (which all start with "\\servername"), except that I had a bug and instead was iterating through the characters of the first network folder path. And that's how I discovered that in windows, "\" means root of the current active drive. And that's also why I value my automatic backup to a NAS twice a day.