9 ms·

How Pixar Almost Lost Toy Story 2 To A Bad Backup

169 points by trevin 14y ago

kenrikm 14y ago
It can't be that simple, Pixar has to have massive distributed systems right? I mean it can't be a single linux system with a single backup, if that's the case you would be safer storing it in Dropbox (Not that you would) since at least then you would have copies on all of your nodes.
- timr 14y ago
  I imagine they're referring not to the rendered frames, but to the models, textures, etc. that the animators were using. And honestly...the larger the system, the more things that can go wrong. Lots of people back up files to RAID arrays, only to find out that their parity drive(s) are hosed at the worst possible moment. The price of a good backup system is eternal diligence.
  - MartinCron 14y ago
    And even then, the entire film is created in storyboard form, and then the voice work is done, and then the final rendering is done. Sure, re-doing parts of the work would be a pain, but they wouldn't have had to get Tim Allen back into the studio.
- Arelius 14y ago
  If you are suggesting that could they store Toy Story II content in Dropbox you A. forget how long ago Toy Story II was (1999) and B. underestimate the size and throughput of all the data that goes into making such a movie. edit: added "could" to make it clear that I'm suggesting that such was an impossibility.
  - cmelbye 14y ago
    Did you just decide to not read the part where he said "Not that you would"? He was making a point, not actually suggesting that they use Dropbox before it was invented to make backups of their multi million dollar movie production.
    - Arelius 14y ago
      "Not that you would" Implies that you could. I was pointing out that they could not, for two rather poignant reasons. I didn't mean to do it offensively, but the point is that syncing and managing the huge amount of data that they have would be difficult and impractical over the internet even today, and this was also 13 years ago. Especially then, maintaining a single "master" copy on a single linux machine was in fact a pretty decent solution to the problem.
    - kenrikm 14y ago
      ^ This. However wow it's been that long? My point was a single linux box seems "Low tech" considering you need a render farm just to produce the movie. Also they were clearly referring to assets and not the rendered film (Hats, sets etc.. getting deleted) which is 1000x worse.
      - Arelius 14y ago
        The infrastructure to maintain authority on terabytes of data on late 90's hardware in a distributed fashion would be a pretty significant amount of development effort. It'd be much easier to just get a giant RAID NAS, and manage it all there, which should in theory be quite sufficient assuming you are keeping proper backups. Many of the render nodes will have copies of the needed data, but the authoritative copy needs to live somewhere.
  - raverbashing 14y ago
    The tech of 1999 is certainly underestimated today Still, in 1999 I had a PII 333MHz and 10GB HD (and Windows 98, sigh) But reading about the history of Pixar, the difficulties (in the first Toy Story) involved the workflow of creating a full-feature film (apart from all the other problems they had) So I'm guessing, since Toy Story 2 was their 3rd feature film, things were still a bit raw (especially given their fast pace)
    - Arelius 14y ago
      How is that underestimated, a world of 10GB Hardrives when that's much less than the base package of dropbox these days. Managing the terabytes of data required for the movie back then was a much more difficult task.
      - raverbashing 14y ago
        What is underestimated is what one could do with it. Sure, 10GB fits in a usb stick today (or better, fits in a 3 year old one) And managing the amount of data was certainly a challenge. If you see the first Toy Story today you can think of that being rendered in real time by a modern graphic card easily. But still, they had what was available and managed to do all kinds of tricks Now see what they could do in 1984 http://www.youtube.com/watch?v=TYbPzyvResc http://www.youtube.com/watch?v=TYbPzyvResc
        Arelius 14y ago
        My comments are specifically about data storage. It was much harder to store terabytes back then.
- windsurfer 14y ago
  Toy story 2 was quite a while ago. I'm sure it's much different now.
- randomdrake 14y ago
  Toy Story 2 was released in 1999. I'm assuming they were working on it quite a bit before then. I think you're overestimating the tech available at that time
- orenjacob 14y ago
  Hi, I'm the "Oren Jacob" from the video. Rendering at that time was distributed across hundreds of CPUs in hundreds of Sun boxes in a renderfarm. The authoritative version of the film's data (models, animation, lighting, sets, shaders, textures, etc..) which was where all rendering pulled data from or verified that it had the most up to date versions, was stored on a single machine. There were several backup strategies for that machine, but they failed. A few reasons are outlined elsewhere in this thread (running past the 4gig limit on tapes) but, if memory serves, other errors were also occurring that involved drives being out of space so error logging wasn't working properly. Making matters worse, after our initial restoral, the crew came back to start working on the show and then, a few days later, we discovered that the restoral we thought was good was, in fact, not. So we had, in effect, split the source tree and now had a bad branch with a week of the crew's work on it, including animation. So we had to untangle the mess as best as we could, which was about 70% script based, and about 30% by hand. Further details are in my post on the Quora thread... (http://www.quora.com/Pixar-Animation-Studios/Did-Pixar-accidentally-delete-Toy-Story-2-during-production/answer/Oren-Jacob http://www.quora.com/Pixar-Animation-Studios/Did-Pixar-accid...)
alecco 14y ago
Video minus blogspam http://www.youtube.com/watch?v=EL_g0tyaIeE http://www.youtube.com/watch?v=EL_g0tyaIeE
Legion 14y ago
Obviously, Ms. Susman's circa-1998 home computer would in no way be able to replicate the level of storage provided by the Sun server cluster and its 4.5 TB disk array (http://www.hoise.com/primeur/99/articles/monthly/AE-PR-09-99-53.html http://www.hoise.com/primeur/99/articles/monthly/AE-PR-09-99...). What was deleted, per Loren Carpenter on Quora, was, "in effect the database(s) containing the master copies of characters, sets, animation, etc. The frames were to be computed from that. A few hundred MB, if I remember." (http://www.quora.com/Pixar-Animation-Studios/Did-Pixar-accidentally-delete-Toy-Story-2-during-production/answer/Loren-Carpenter http://www.quora.com/Pixar-Animation-Studios/Did-Pixar-accid...) A little more detail than provided in the video, where they just called it, "the film", as if the movie's entirety of data were both deleted and duplicated on that home PC.
- dclowd9901 14y ago
  Thank you for explaining this. It was literally the first thing that I was wondering.
- orenjacob 14y ago
  It is worth noting that Galyn's home machine, and the one that I drove over to get with her and return to Pixar as depicted in the video in question, was in fact an SGI. Not your average "home computer" by any stretch. Although you are correct, there was nowhere near 4.5 terabyte of drive space in that Indigo II or Octane (I forget what she had in her house specifically, although I can ask her if people really want to know). The sum of all the source files (models, animation, shaders, lighting, sets, etc..) was probably in the hundreds-of-megabytes range at that point in time as Loren pointed out in his comments on Quora. I've also added a lot more details to that Quora thread too (http://www.quora.com/Pixar-Animation-Studios/Did-Pixar-accidentally-delete-Toy-Story-2-during-production/answer/Oren-Jacob http://www.quora.com/Pixar-Animation-Studios/Did-Pixar-accid...) if folks want to read more about the whole thing, although it was 14 years ago so my memory is a little foggy on some of the details at this point in time.
  - deleted 14y ago
    [deleted]
radicalbyte 14y ago
Someone had ran RM * on their shared storage, deleting the file system. They were "saved" by someone having a copy of the movie on their PC at home. The data wasn't gone (only the file system was nuked), but would have required specialist intervention to restore. Or a hacker with the right tools (assuming such tools exist for the *nix file systems - they certainly do for fat32 and NTFS)..
- azelfrath 14y ago
  I would assume this would be standard forensics. Might not be tools for their particular file format, but it would be possible to carve the data.
- Zirro 14y ago
  "Someone had ran RM * on their shared storage, deleting the file system." Doesn't this command, like any other delete-command, require top privileges (admin/root) that are unavailable in normal situations?
  - mcpherrinm 14y ago
    Presumably the people working on the film had the ability to edit the assets, and thus could delete them.
    - Zirro 14y ago
      I'm still under the impression that modifying and completely removing are different things, but perhaps it was a sloppy configuration or that they put too much trust in their employees.
      - sliverstorm 14y ago
        Depends on the filesystem. Traditional unix permissions are simply r, w, x.
        14y ago
nyellin 14y ago
Where is this true? It was most likely "sudo rm -rf .*" which will actually go BACKWARDS up the tree as well. Edit: At the very least, this doesn't work on OS X 10.5, Ubuntu 10.04 (server edition), and CentOS 5.8. For example: $ mkdir -p foo/bar $ cd foo/bar $ rm -rf .. rm: "." and ".." may not be removed
- archivator 14y ago
  Definitely true on Linux with bash - .* will expand to .. as well as all hidden files/folders.
- Flimm 14y ago
  The shell expands .* to all directories beginning with a dot, including the two special hard links "." and ".." . See for yourself by running: echo .*
  - deleted 14y ago
    [deleted]
- mikeash 14y ago
  'echo .*' and notice how the second entry is .., the parent directory. It won't go up the tree indefinitely, but it will go up one level.
- CrLf 14y ago
  Well, that happened to me at least once on Linux, ten years or so ago. I don't remember if it goes backwards up the tree or just one level (I think it's the latter), or if the current 'rm' protects against this or not. Imagine you place yourself in '/tmp' to throw away all those pesky '.' directories and files that applications leave there. You run that command, it matches '..' and your whole system is gone (btw, 'rm -rf .??*' avoids this).
  - ars 14y ago
    Both are true: It will only go up one level, and current rm has protection that will not let it delete the root.
ja27 14y ago
I did this to myself in college, thankfully on machines where I didn't have most of my stuff. I figured it was an annoyance and put in a restore request to the support group. Days went by and I was getting frustrated that they still hadn't gotten around to it. Finally I caught the head of the support group in the hall and asked about it. "Well, the good news is that we're getting a new backup system." Long pause. "The bad news is that your files are at about the third gigabyte of a two gigabyte tape." Ever since, I'm always frustrated by people that assume things are backed up. If you don't test your backups, you don't have backups.
- bitops 14y ago
  > If you don't test your backups, you don't have backups. I couldn't agree more! It's amazing how often the question "we take backups, right?" produces "uuhh..". And yes, if you don't test them, they're worthless. Because the rule is, if you haven't tested them, they won't work when the shit's really hitting the fan.
  - pstuart 14y ago
    Back in the day we would take the days backup tapes and restore them to a test system that was also available as a "playground".
  - franze 14y ago
    i especially love the (existing) backups that can't be actually used to bring the system "back up" .... experienced this more than once.
  - bemmu 14y ago
    How should I test my backups? I have a projects folder, which is on Dropbox and additionally on Time Machine. I can confirm they are really in both by browsing them. Anything more I should do?
    - PeterisP 14y ago
      Take a fresh, empty system (a VM if you lack hardware); restore data from backups to it; check if you can do your work. It may be that your projects folder really is backed up, but that it relies on some stuff that is outside the folder for whatever reason - for example, a data file that didn't fit in dropbox, or some changes to config files outside the folder (where noone remembers exactly what needed to be changed); or a connection that relies on a private key that's on your computer, gets used automagically, but is not backed up. I mean, it's probably overkill in your case - most likely your project source is quite safe; but for a working production system it's an entirely another matter.
serverascode 14y ago
After about 10 years of not doing "rm -rf *" I finally did manage to hose an entire system when I entered the wrong directory (/) for script I had written to randomly alter files. 'Course it was on a test vm, but it was still pretty amazing to watch what happens when 10% of the files on a system had random data added to them, or were randomly removed. :)
egypturnash 14y ago
Oh right, seeing this in the morning browse reminded me I was gonna test my offsite backups, but I was in bed with the iPad at the time. pokes Backblaze and grabs a couple of random files It works! Yay.
larrys 14y ago
It wasn't just a bad backup. It was a bad backup "system". One that didn't involve testing the backups obviously or even any redundancy. I guess the sysadmin never read the story (urban legend?) about the guy in Sweden who brought his backup tapes home and erased them on the car's electric seats. The comment on the cartoon "I remember thinking this is really bad we're going to get our backups" comes from someone who obviously does not have the appropriate fear of what can happen to even begin to understand a good backup system. I'm guessing she is much more careful with her children and anticipates things in advance. As far as "the backup being bad" where were they stored? Were they stored offsite?
- excuse-me 14y ago
  Probably written wrong not degraded. It's very easy to say the backup should be tested, there should be mirrored recovery systems etc. but this is a movie, even in a studio system this is a one shot deal, the systems and even the hardware will be disposed of after the master is cut,the people will leave for other jobs etc. Now in the middle of crunch to make the movie - tell the producers you are going to double the HW budget for a test system to test the backups and you are going to stop doing any new work until a backup system is built and tested. Personally I'm amazed they had any sort of backup beyond personal copies on animators machines.
  - bdunbar 14y ago
    Probably written wrong not degraded. That was my thought. Heck, it happened to me once. Combination of a SCSI cable knocked loose, but not visibly so, and the brand new sysamdin (me in both cases) knowing enough to head and tail the backup log file looking for 'start' and 'complete' but not knowing to look in the middle for what was actually happening. Lost a month of data in the (thankfully) little used legacy ERP system we used. If the entire plant had still been using it it would have been a disaster, not an inconvenience.
  - Drbble 14y ago
    Pixar is a movie studio that developed mpviemaking technology over many years. No one is talking about backing up Tom Hanks in case he keels over.
jwallaceparker 14y ago
I'm curious to hear more about users' standard backup systems for life/work (other that a version control system for code). Any thoughts?
- nfriedly 14y ago
  Google has my email, Dropbox has my taxes and a few other important documents, wedding photos are also on a family member's computer, and svn / github takes care of my code for work / play. Nothing else I have is that important. Lots of games and recorded TV shows. Either I already finished it, or I'm probably not going to. I could loose my desktop or my laptop and only be out time. If both died at once I might loose a few important things, but nothing too life-shattering. In general, I can pick up a brand new computer and be productive by the end of the day.
jwallaceparker 14y ago
I'm curious to hear more about users' standard backup systems for life/work (other that a version control system for code). Any thoughts?
rootbear 14y ago
I was there at the time this happened. They were doing backups, but not testing them. The version of tar being used, from IRIX, I think, used 32-bit file offsets and couldn't back up more than 4GB of data to one file. The TS2 models and animation files took up more than that and it overflowed the maximum file size. The moral is that untested backups aren't backups.
- excuse-me 14y ago
  Same era we had a backup that wrote to a DAT tape "/dev/nrst0", Somebody mis-typed it and the backups were all going to a file "/dev/nstr0" - we only noticed when somebody claimed they could read the backup with the tape ejected ! SunOS version mid 1990s - you had to be root to run the backup script AND you could create files in /dev without warning AND / and /home were in the same partition so it had lots of space
- DeepDuh 14y ago
  If she wouldn't have had the backup in her house, couldn't you have simply restored the files from the original disks? Of course you would have to rebuild the folder structure somehow, but that would be the work of some days, not months, right?
kevinchen 14y ago
Regardless of what happened, I don't think Toy Story 2 really would have been lost. The writers and animators would at least be able to recreate the general idea from what they remembered.
- nikatwork 14y ago
  I've lost creative works to bad backups - even if you recreate the work immediately, it's Never The Same!
  - pwthornton 14y ago
    Yes, the creative process often works in a way that either you do the idea that popped into your head when it comes in or it never comes to fruition. You can't recreate the moment when you created something.
  - kabdib 14y ago
    I once lost two weeks of programming to an errant delete. I recreated it in about two days, and it was better. Obviously, artwork != code. But I find a second pass nearly always improves things.
pwthornton 14y ago
My how far we've come. Today, many home owners have good backups. But a little more than 10 years ago, one of the most important movies of the last few decades was almost lost forever. I imagine that Pixar has a lot of backups and some of them are offsite. Back in the late 1990s, I don't think offsite backups of a project like this would work.
orenjacob 14y ago
My memory is foggy (it was 13 or 14 years ago) but to share a little bit of the technical landscape at that time with folks reading this thread... I think we set things up giving her a full tree with her SGI machine on the netwrok, physically in the building. Then we sent the machine home with her and incrementally updated it weekly via rdist over an ISDN line. I wish we had Dropbox back then!
simonh 14y ago
A few months into my first job a department called because their PC wouldn't boot, the hard disk was dead. They took tape backups every day, so I checked the last tape but it was blank. I asked if they backed up every day and said yes, and said that for the last two weeks or so the backups had taken only a few seconds, where they used to take about an hour. They were very pleased because it was such a tedious job. They had one weeks worth of tapes and they were all blank. Turns out the tape drive had gone faulty a few weeks before the hard drive failed, just long enough to go through a full cycle of all their backup tapes.
tutu12777 14y ago
Dkel
gouranga 14y ago
I watched some poor guy at an agency I worked for slide into the drive the single CDRW that his life's work was sitting on. A few whizzes later a large explosion shook the room from the drive and bits of plastic went flying all over the room from the drive (it was one of those shitty pioneer slot loaders). He literally, despite being 30, cried his eyes out for about an hour. That was when I learned not to fuck around with data security and backups... I wish everyone this experience at least once but without the pain of losing so much. It's really important to always have the backup monkey on your back.
- andywood 14y ago
  When I was 16, I lost a hard drive with 2 years of game development work on it. I eventually rebuilt the game much bigger and better, and shipped it, but not before I cried for three days straight. Now I never leave anything un-backed up that I would be disappointed about losing. Fortunately for us, storage is far cheaper now than it was when I was a kid.
krosaen 14y ago
"And then, some months later, Pixar rewrote the film from almost the ground up, and we made ToyStory2 again. That rewritten film was the one you saw in theatres and that you can watch now on BluRay." http://www.quora.com/Pixar-Animation-Studios/Did-Pixar-accidentally-delete-Toy-Story-2-during-production http://www.quora.com/Pixar-Animation-Studios/Did-Pixar-accid...
gregcohn 14y ago
An interesting dimension of this that other commenters haven't noted is that keeping a home backup of this scope would be so clearly against policy at most corporate technology companies as to be a termination-worthy offense.

169 points by trevin 14y ago

It can't be that simple, Pixar has to have massive distributed systems right? I mean it can't be a single linux system with a single backup, if that's the case you would be safer storing it in Dropbox (Not that you would) since at least then you would have copies on all of your nodes.

timr 14y ago
I imagine they're referring not to the rendered frames, but to the models, textures, etc. that the animators were using. And honestly...the larger the system, the more things that can go wrong. Lots of people back up files to RAID arrays, only to find out that their parity drive(s) are hosed at the worst possible moment. The price of a good backup system is eternal diligence.
- MartinCron 14y ago
  And even then, the entire film is created in storyboard form, and then the voice work is done, and then the final rendering is done. Sure, re-doing parts of the work would be a pain, but they wouldn't have had to get Tim Allen back into the studio.
Arelius 14y ago
If you are suggesting that could they store Toy Story II content in Dropbox you A. forget how long ago Toy Story II was (1999) and B. underestimate the size and throughput of all the data that goes into making such a movie. edit: added "could" to make it clear that I'm suggesting that such was an impossibility.
- cmelbye 14y ago
  Did you just decide to not read the part where he said "Not that you would"? He was making a point, not actually suggesting that they use Dropbox before it was invented to make backups of their multi million dollar movie production.
  - Arelius 14y ago
    "Not that you would" Implies that you could. I was pointing out that they could not, for two rather poignant reasons. I didn't mean to do it offensively, but the point is that syncing and managing the huge amount of data that they have would be difficult and impractical over the internet even today, and this was also 13 years ago. Especially then, maintaining a single "master" copy on a single linux machine was in fact a pretty decent solution to the problem.
  - kenrikm 14y ago
    ^ This. However wow it's been that long? My point was a single linux box seems "Low tech" considering you need a render farm just to produce the movie. Also they were clearly referring to assets and not the rendered film (Hats, sets etc.. getting deleted) which is 1000x worse.
    - Arelius 14y ago
      The infrastructure to maintain authority on terabytes of data on late 90's hardware in a distributed fashion would be a pretty significant amount of development effort. It'd be much easier to just get a giant RAID NAS, and manage it all there, which should in theory be quite sufficient assuming you are keeping proper backups. Many of the render nodes will have copies of the needed data, but the authoritative copy needs to live somewhere.
- raverbashing 14y ago
  The tech of 1999 is certainly underestimated today Still, in 1999 I had a PII 333MHz and 10GB HD (and Windows 98, sigh) But reading about the history of Pixar, the difficulties (in the first Toy Story) involved the workflow of creating a full-feature film (apart from all the other problems they had) So I'm guessing, since Toy Story 2 was their 3rd feature film, things were still a bit raw (especially given their fast pace)
  - Arelius 14y ago
    How is that underestimated, a world of 10GB Hardrives when that's much less than the base package of dropbox these days. Managing the terabytes of data required for the movie back then was a much more difficult task.
    - raverbashing 14y ago
      What is underestimated is what one could do with it. Sure, 10GB fits in a usb stick today (or better, fits in a 3 year old one) And managing the amount of data was certainly a challenge. If you see the first Toy Story today you can think of that being rendered in real time by a modern graphic card easily. But still, they had what was available and managed to do all kinds of tricks Now see what they could do in 1984 http://www.youtube.com/watch?v=TYbPzyvResc http://www.youtube.com/watch?v=TYbPzyvResc
      - Arelius 14y ago
        My comments are specifically about data storage. It was much harder to store terabytes back then.
windsurfer 14y ago
Toy story 2 was quite a while ago. I'm sure it's much different now.
randomdrake 14y ago
Toy Story 2 was released in 1999. I'm assuming they were working on it quite a bit before then. I think you're overestimating the tech available at that time
orenjacob 14y ago
Hi, I'm the "Oren Jacob" from the video. Rendering at that time was distributed across hundreds of CPUs in hundreds of Sun boxes in a renderfarm. The authoritative version of the film's data (models, animation, lighting, sets, shaders, textures, etc..) which was where all rendering pulled data from or verified that it had the most up to date versions, was stored on a single machine. There were several backup strategies for that machine, but they failed. A few reasons are outlined elsewhere in this thread (running past the 4gig limit on tapes) but, if memory serves, other errors were also occurring that involved drives being out of space so error logging wasn't working properly. Making matters worse, after our initial restoral, the crew came back to start working on the show and then, a few days later, we discovered that the restoral we thought was good was, in fact, not. So we had, in effect, split the source tree and now had a bad branch with a week of the crew's work on it, including animation. So we had to untangle the mess as best as we could, which was about 70% script based, and about 30% by hand. Further details are in my post on the Quora thread... (http://www.quora.com/Pixar-Animation-Studios/Did-Pixar-accidentally-delete-Toy-Story-2-during-production/answer/Oren-Jacob http://www.quora.com/Pixar-Animation-Studios/Did-Pixar-accid...)

alecco 14y ago

Video minus blogspam http://www.youtube.com/watch?v=EL_g0tyaIeE http://www.youtube.com/watch?v=EL_g0tyaIeE

Legion 14y ago

Obviously, Ms. Susman's circa-1998 home computer would in no way be able to replicate the level of storage provided by the Sun server cluster and its 4.5 TB disk array (http://www.hoise.com/primeur/99/articles/monthly/AE-PR-09-99-53.html http://www.hoise.com/primeur/99/articles/monthly/AE-PR-09-99...). What was deleted, per Loren Carpenter on Quora, was, "in effect the database(s) containing the master copies of characters, sets, animation, etc. The frames were to be computed from that. A few hundred MB, if I remember." (http://www.quora.com/Pixar-Animation-Studios/Did-Pixar-accidentally-delete-Toy-Story-2-during-production/answer/Loren-Carpenter http://www.quora.com/Pixar-Animation-Studios/Did-Pixar-accid...) A little more detail than provided in the video, where they just called it, "the film", as if the movie's entirety of data were both deleted and duplicated on that home PC.

dclowd9901 14y ago
Thank you for explaining this. It was literally the first thing that I was wondering.
orenjacob 14y ago
It is worth noting that Galyn's home machine, and the one that I drove over to get with her and return to Pixar as depicted in the video in question, was in fact an SGI. Not your average "home computer" by any stretch. Although you are correct, there was nowhere near 4.5 terabyte of drive space in that Indigo II or Octane (I forget what she had in her house specifically, although I can ask her if people really want to know). The sum of all the source files (models, animation, shaders, lighting, sets, etc..) was probably in the hundreds-of-megabytes range at that point in time as Loren pointed out in his comments on Quora. I've also added a lot more details to that Quora thread too (http://www.quora.com/Pixar-Animation-Studios/Did-Pixar-accidentally-delete-Toy-Story-2-during-production/answer/Oren-Jacob http://www.quora.com/Pixar-Animation-Studios/Did-Pixar-accid...) if folks want to read more about the whole thing, although it was 14 years ago so my memory is a little foggy on some of the details at this point in time.
- deleted 14y ago
  [deleted]

radicalbyte 14y ago

Someone had ran RM * on their shared storage, deleting the file system. They were "saved" by someone having a copy of the movie on their PC at home. The data wasn't gone (only the file system was nuked), but would have required specialist intervention to restore. Or a hacker with the right tools (assuming such tools exist for the *nix file systems - they certainly do for fat32 and NTFS)..

azelfrath 14y ago
I would assume this would be standard forensics. Might not be tools for their particular file format, but it would be possible to carve the data.
Zirro 14y ago
"Someone had ran RM * on their shared storage, deleting the file system." Doesn't this command, like any other delete-command, require top privileges (admin/root) that are unavailable in normal situations?
- mcpherrinm 14y ago
  Presumably the people working on the film had the ability to edit the assets, and thus could delete them.
  - Zirro 14y ago
    I'm still under the impression that modifying and completely removing are different things, but perhaps it was a sloppy configuration or that they put too much trust in their employees.
    - sliverstorm 14y ago
      Depends on the filesystem. Traditional unix permissions are simply r, w, x.
      - 14y ago

nyellin 14y ago

Where is this true? It was most likely "sudo rm -rf .*" which will actually go BACKWARDS up the tree as well. Edit: At the very least, this doesn't work on OS X 10.5, Ubuntu 10.04 (server edition), and CentOS 5.8. For example: $ mkdir -p foo/bar $ cd foo/bar $ rm -rf .. rm: "." and ".." may not be removed

archivator 14y ago
Definitely true on Linux with bash - .* will expand to .. as well as all hidden files/folders.
Flimm 14y ago
The shell expands .* to all directories beginning with a dot, including the two special hard links "." and ".." . See for yourself by running: echo .*
- deleted 14y ago
  [deleted]
mikeash 14y ago
'echo .*' and notice how the second entry is .., the parent directory. It won't go up the tree indefinitely, but it will go up one level.
CrLf 14y ago
Well, that happened to me at least once on Linux, ten years or so ago. I don't remember if it goes backwards up the tree or just one level (I think it's the latter), or if the current 'rm' protects against this or not. Imagine you place yourself in '/tmp' to throw away all those pesky '.' directories and files that applications leave there. You run that command, it matches '..' and your whole system is gone (btw, 'rm -rf .??*' avoids this).
- ars 14y ago
  Both are true: It will only go up one level, and current rm has protection that will not let it delete the root.

ja27 14y ago

I did this to myself in college, thankfully on machines where I didn't have most of my stuff. I figured it was an annoyance and put in a restore request to the support group. Days went by and I was getting frustrated that they still hadn't gotten around to it. Finally I caught the head of the support group in the hall and asked about it. "Well, the good news is that we're getting a new backup system." Long pause. "The bad news is that your files are at about the third gigabyte of a two gigabyte tape." Ever since, I'm always frustrated by people that assume things are backed up. If you don't test your backups, you don't have backups.

bitops 14y ago
> If you don't test your backups, you don't have backups. I couldn't agree more! It's amazing how often the question "we take backups, right?" produces "uuhh..". And yes, if you don't test them, they're worthless. Because the rule is, if you haven't tested them, they won't work when the shit's really hitting the fan.
- pstuart 14y ago
  Back in the day we would take the days backup tapes and restore them to a test system that was also available as a "playground".
- franze 14y ago
  i especially love the (existing) backups that can't be actually used to bring the system "back up" .... experienced this more than once.
- bemmu 14y ago
  How should I test my backups? I have a projects folder, which is on Dropbox and additionally on Time Machine. I can confirm they are really in both by browsing them. Anything more I should do?
  - PeterisP 14y ago
    Take a fresh, empty system (a VM if you lack hardware); restore data from backups to it; check if you can do your work. It may be that your projects folder really is backed up, but that it relies on some stuff that is outside the folder for whatever reason - for example, a data file that didn't fit in dropbox, or some changes to config files outside the folder (where noone remembers exactly what needed to be changed); or a connection that relies on a private key that's on your computer, gets used automagically, but is not backed up. I mean, it's probably overkill in your case - most likely your project source is quite safe; but for a working production system it's an entirely another matter.

serverascode 14y ago

After about 10 years of not doing "rm -rf *" I finally did manage to hose an entire system when I entered the wrong directory (/) for script I had written to randomly alter files. 'Course it was on a test vm, but it was still pretty amazing to watch what happens when 10% of the files on a system had random data added to them, or were randomly removed. :)

egypturnash 14y ago

Oh right, seeing this in the morning browse reminded me I was gonna test my offsite backups, but I was in bed with the iPad at the time. pokes Backblaze and grabs a couple of random files It works! Yay.

larrys 14y ago

It wasn't just a bad backup. It was a bad backup "system". One that didn't involve testing the backups obviously or even any redundancy. I guess the sysadmin never read the story (urban legend?) about the guy in Sweden who brought his backup tapes home and erased them on the car's electric seats. The comment on the cartoon "I remember thinking this is really bad we're going to get our backups" comes from someone who obviously does not have the appropriate fear of what can happen to even begin to understand a good backup system. I'm guessing she is much more careful with her children and anticipates things in advance. As far as "the backup being bad" where were they stored? Were they stored offsite?

excuse-me 14y ago
Probably written wrong not degraded. It's very easy to say the backup should be tested, there should be mirrored recovery systems etc. but this is a movie, even in a studio system this is a one shot deal, the systems and even the hardware will be disposed of after the master is cut,the people will leave for other jobs etc. Now in the middle of crunch to make the movie - tell the producers you are going to double the HW budget for a test system to test the backups and you are going to stop doing any new work until a backup system is built and tested. Personally I'm amazed they had any sort of backup beyond personal copies on animators machines.
- bdunbar 14y ago
  Probably written wrong not degraded. That was my thought. Heck, it happened to me once. Combination of a SCSI cable knocked loose, but not visibly so, and the brand new sysamdin (me in both cases) knowing enough to head and tail the backup log file looking for 'start' and 'complete' but not knowing to look in the middle for what was actually happening. Lost a month of data in the (thankfully) little used legacy ERP system we used. If the entire plant had still been using it it would have been a disaster, not an inconvenience.
- Drbble 14y ago
  Pixar is a movie studio that developed mpviemaking technology over many years. No one is talking about backing up Tom Hanks in case he keels over.

jwallaceparker 14y ago

I'm curious to hear more about users' standard backup systems for life/work (other that a version control system for code). Any thoughts?

nfriedly 14y ago
Google has my email, Dropbox has my taxes and a few other important documents, wedding photos are also on a family member's computer, and svn / github takes care of my code for work / play. Nothing else I have is that important. Lots of games and recorded TV shows. Either I already finished it, or I'm probably not going to. I could loose my desktop or my laptop and only be out time. If both died at once I might loose a few important things, but nothing too life-shattering. In general, I can pick up a brand new computer and be productive by the end of the day.

jwallaceparker 14y ago

I'm curious to hear more about users' standard backup systems for life/work (other that a version control system for code). Any thoughts?

rootbear 14y ago

I was there at the time this happened. They were doing backups, but not testing them. The version of tar being used, from IRIX, I think, used 32-bit file offsets and couldn't back up more than 4GB of data to one file. The TS2 models and animation files took up more than that and it overflowed the maximum file size. The moral is that untested backups aren't backups.

excuse-me 14y ago
Same era we had a backup that wrote to a DAT tape "/dev/nrst0", Somebody mis-typed it and the backups were all going to a file "/dev/nstr0" - we only noticed when somebody claimed they could read the backup with the tape ejected ! SunOS version mid 1990s - you had to be root to run the backup script AND you could create files in /dev without warning AND / and /home were in the same partition so it had lots of space
DeepDuh 14y ago
If she wouldn't have had the backup in her house, couldn't you have simply restored the files from the original disks? Of course you would have to rebuild the folder structure somehow, but that would be the work of some days, not months, right?

kevinchen 14y ago

Regardless of what happened, I don't think Toy Story 2 really would have been lost. The writers and animators would at least be able to recreate the general idea from what they remembered.

nikatwork 14y ago
I've lost creative works to bad backups - even if you recreate the work immediately, it's Never The Same!
- pwthornton 14y ago
  Yes, the creative process often works in a way that either you do the idea that popped into your head when it comes in or it never comes to fruition. You can't recreate the moment when you created something.
- kabdib 14y ago
  I once lost two weeks of programming to an errant delete. I recreated it in about two days, and it was better. Obviously, artwork != code. But I find a second pass nearly always improves things.

pwthornton 14y ago

My how far we've come. Today, many home owners have good backups. But a little more than 10 years ago, one of the most important movies of the last few decades was almost lost forever. I imagine that Pixar has a lot of backups and some of them are offsite. Back in the late 1990s, I don't think offsite backups of a project like this would work.

orenjacob 14y ago

My memory is foggy (it was 13 or 14 years ago) but to share a little bit of the technical landscape at that time with folks reading this thread... I think we set things up giving her a full tree with her SGI machine on the netwrok, physically in the building. Then we sent the machine home with her and incrementally updated it weekly via rdist over an ISDN line. I wish we had Dropbox back then!

simonh 14y ago

A few months into my first job a department called because their PC wouldn't boot, the hard disk was dead. They took tape backups every day, so I checked the last tape but it was blank. I asked if they backed up every day and said yes, and said that for the last two weeks or so the backups had taken only a few seconds, where they used to take about an hour. They were very pleased because it was such a tedious job. They had one weeks worth of tapes and they were all blank. Turns out the tape drive had gone faulty a few weeks before the hard drive failed, just long enough to go through a full cycle of all their backup tapes.

tutu12777 14y ago

Dkel

gouranga 14y ago

I watched some poor guy at an agency I worked for slide into the drive the single CDRW that his life's work was sitting on. A few whizzes later a large explosion shook the room from the drive and bits of plastic went flying all over the room from the drive (it was one of those shitty pioneer slot loaders). He literally, despite being 30, cried his eyes out for about an hour. That was when I learned not to fuck around with data security and backups... I wish everyone this experience at least once but without the pain of losing so much. It's really important to always have the backup monkey on your back.

andywood 14y ago
When I was 16, I lost a hard drive with 2 years of game development work on it. I eventually rebuilt the game much bigger and better, and shipped it, but not before I cried for three days straight. Now I never leave anything un-backed up that I would be disappointed about losing. Fortunately for us, storage is far cheaper now than it was when I was a kid.

krosaen 14y ago

"And then, some months later, Pixar rewrote the film from almost the ground up, and we made ToyStory2 again. That rewritten film was the one you saw in theatres and that you can watch now on BluRay." http://www.quora.com/Pixar-Animation-Studios/Did-Pixar-accidentally-delete-Toy-Story-2-during-production http://www.quora.com/Pixar-Animation-Studios/Did-Pixar-accid...

gregcohn 14y ago

An interesting dimension of this that other commenters haven't noted is that keeping a home backup of this scope would be so clearly against policy at most corporate technology companies as to be a termination-worthy offense.