14 ms·
A baby saved ‘Toy Story 2’ from near complete deletion (2021)
- peter303 5y agoI have done forensic reconstruction of UNIX files. The directory node list is erased/lost, but the raw data remains on disk. This assumes the data has a sequential structure. The article didnt say whether this was the model data or the rendered data.
- blhack 5y agoThe data was saved because a mother was working from home with her baby, and was keeping a backup of the files there.
- CharlesW 5y ago"Pixar Studio Stories: The Movie Vanishes", narrated by Oren Jacob and Galyn Susman: https://www.youtube.com/watch?v=8dhp_20j0Ys https://www.youtube.com/watch?v=8dhp_20j0Ys "Did Pixar accidentally delete Toy Story 2 during production?", Quora answer by Oren Jacob: https://www.quora.com/Pixar-company/Did-Pixar-accidentally-delete-Toy-Story-2-during-production/answer/Oren-Jacob https://www.quora.com/Pixar-company/Did-Pixar-accidentally-d...
- peter303 5y agoHome computers where pretty dinky and broadband slow at the time of Toy Story 2. I wonder how they'd get 90% of the movie transferred home and saved to disk/tape?
- Melatonic 5y agoDefinitely the "send a production assistant with an external drive(s) driving their old beater car" method :-D
- Keyframe 5y ago99.999% chance that 'home computer' was an SGI box.
- rootbear 5y agoIt was.I don't know exactly what, but we had a lot of Octanes.
- Keyframe 5y agoYeah, there was nothing else for workstation in VFX and 3D back then in film. Occasional sun box for rendering (Pixar might've had more than a few vs origins at ILM). Amiga's and 3dsmax for TV on PC and intergraph PC boxes.. Maya was very new after 9 versions of PA and TDI explore, Softimage just peaked into windows.. you guys did your own thing with Marionette and whatnot. Exciting times.
- orenjacob 5y agoHi folks, I'm the Oren mentioned in the story(ies) above. This comes up on HN every year or two. Quick reference to lots more detail posted here -> https://www.quora.com/Did-Pixar-accidentally-delete-Toy-Story-2-during-production https://www.quora.com/Did-Pixar-accidentally-delete-Toy-Stor...
- ldamerow 5y agoYou beat me to it. :D
- Melatonic 5y agoWhy not just copy all the data while at her house? Seems like moving the machine was an unnecessary risk. Bring over a box with some large drives and just let it run overnight?
- orenjacob 5y agoOK, yeah, that would have been another approach. Perhaps the emotion of the moment motivated us to get the machine back on-prem as soon as possible.
- quickthrower2 5y agoThe backup rule of 3 worked out then, if accidentally.
- w0mbat 5y agoThis story has been told many times and this account is not the most complete. Pixar did recover the data from a home office setup and complete the movie, but Pixar is so perfectionist about story-telling that the creative heads decided the end result wasn't good enough. The story was rewritten and the movie changed so much that almost none of the recovered data was in the version that was finally released.
- orenjacob 5y agoAlso, here's the HackerNews thread from 2012 that answers a bunch of additional questions too... https://news.ycombinator.com/item?id=3972798 https://news.ycombinator.com/item?id=3972798
- stereoradonc 5y agoThis was cringe. I don't know how it even got traction. Someone was on maternity leave with access to offline backups, which doesn't mean a "baby saved" the movie. Misleading headline.
- killingtime74 5y agoReporters got to get paid
- achow 5y agoThis is from 1998. Better writeup (2012).. The command that had been run was most likely ‘rm -r -f *’, which—roughly speaking—commands the system to begin removing every file below the current directory. This is commonly used to clear out a subset of unwanted files. https://thenextweb.com/news/how-pixars-toy-story-2-was-deleted-twice-once-by-technology-and-again-for-its-own-good https://thenextweb.com/news/how-pixars-toy-story-2-was-delet...
- lamontcg 5y ago"Hey I just force pushed an old main branch, accidentally, did anyone do a pull earlier today so we can restore?" Kids these days who have branch protection turned on won't get this joke.
- chrismorgan 5y agoIf you’re talking about Git, then as a rule of thumb you’ve got at least 30 days before anything will actually be deleted: you can retrieve what was lost from the reflog. You could start with something like `git checkout "main@{4 hours ago}"`, then delve further into the reflog itself if that doesn’t produce exactly what you want.
- watersb 5y agoWhy oh why do people never practice restoring things? We all treat backups like open loop: billion-dollar project? Meh, just throw it over the wall and hope it lands somewhere. I harp on this every time it comes up. Ugh.
- globular-toast 5y agoThat was disappointing. I was expecting something like a baby crawled over a cable effectively airgapping a system before a disaster happened. The whole story is just "they deleted everything but luckily had a backup".
- alisonkisk 5y ago
- adrianomartins 5y agoWow, this makes me think, do animation teams don't really use a version control system to have multiple people contributing to the same movie at the same time? Does anybody knows how that works?
- rkapsoro 5y agoIIRC perforce is (was?) commonly used in teams with workflows involving large assets.
- cecilpl2 5y agoPerforce is still the defacto standard in the video game industry where it's not uncommon for a game's source assets to run into the tens of terabytes. That said, Toy Story 2 was developed in the late 90s, and while Perforce existed then I don't know how popular it was.
- tbabb 5y agoPixar used RCS at the time. Problem is, when you run `rm -rf /`, that deletes the RCS directories as well.
- doublerabbit 5y agoNot really. With multiple animators working on files simultaneously, its kind of hard. Sadly my animation knowledge has left me since I left the studio working gig three years ago. But we did create content for netflix and many animators came up sheepishly because they deleted a folder for it to be restored. It's not as uncommon as you think. FWIW, the live archive/backup server was called Dumbo. Which were 3x 4u Supermacho chassis with over 1.2PB in drives served over 10Gbit to each workstation connected to 1Gbit running CentOS 5. I dropped the new chassis while racking once and is partly the reason to why I lost my job :/
- doublerabbit 5y agoCan't edit my post, but for clarity. Im wrong on the line "Not really. With multiple animators working on files simultaneously, its kind of hard." see comments above.
- abotsis 5y agoThis seems to be less about how a baby saved the day, and more about how an employee exfiltrated data from her employer. As a cybersecurity guy at a large financial institution, I wonder what Pixar’s policy was about the matter then, and what impact it had on the same policy today.
- ehPReth 5y agoFrom what I recall from another telling of the story Pixar was the one who set her up with a home computer & storage system for that purpose - since the storage and computing power needed was higher than what most people would have at home - but I could be misremembering
- ckozlowski 5y agoYou're on the right track. As told in the Next Web story, the workstation in question was a Silicon Graphics Octane or there abouts. These were incredibly expensive and were provided by the company. In addition, according to the story, they were the ones who set up the file sync for her to be able to work from home. LGR and RMC have both done videos on the SGI Octane and are worth watching: https://www.youtube.com/watch?v=ZDxLa6P6exc https://www.youtube.com/watch?v=ZDxLa6P6exc, https://www.youtube.com/watch?v=VHsA8iq4N0s https://www.youtube.com/watch?v=VHsA8iq4N0s Edit: It was Retro Man Cave, not Adrian's Digital Basement as originally written.
- adl 5y agoThe article is wrong, it wasn't her 'personal computer' it was a Pixar Workstation that was at her home, not the same thing.
- abotsis 5y agoWhy this got downvoted a bit is beyond me, I’ll never understand hn. Oren, who worked at Pixar recounts the story, and said “She has an SGI machine (Indigo? Indigo 2?) Those were the same machines we had at all our desktops to run the animation system and work on the film, which is what she was doing. Yes, it was against the rules, but we did it anyway, and it saved the movie in the end.” https://www.quora.com/Did-Pixar-accidentally-delete-Toy-Story-2-during-production/answer/Oren-Jacob?ch=17&oid=1215894 https://www.quora.com/Did-Pixar-accidentally-delete-Toy-Stor...
- Kaotique 5y agoIf you follow the logic of the title of this article the new policy was that there should always be a mother with a baby on the team.
- tagoregrtst 5y agoNo. Instead you should always treat mothers, especially new moms, with extra respect and care. They have a lot to deal with.
- fulafel 5y agoThis sounds like there's a whole other story in the aftremath, would be interesting to hear how they dealt with it (deciding between dropping scenes vs reproducing): > Jacob admits that about 10% of the film’s files were never retrieved and are gone forever, but they were able to make the film work without the scenes. Also, this seems to be a second hand rereporting, the linked article has a little more detail (https://thenextweb.com/news/how-pixars-toy-story-2-was-deleted-twice-once-by-technology-and-again-for-its-own-good https://thenextweb.com/news/how-pixars-toy-story-2-was-delet...) It also has the real beef of the story: > Pixar, at the time, did not continuously test their backups. This is where the trouble started, because the backups were stored on a tape drive and as the files hit 4 gigabytes in size, the maximum size of the file was met. The error log, which would have told the system administrators about the full drive, was also located on the full volume, and was zero bytes in size.
- rootbear 5y agoThe linked article is much better. This event happened right about the time I left Pixar and I heard about it long ago. In the end, much of what was recovered from Galyn's workstation wasn't used. Some time after this event, it was decided that the film was not working and a huge rewrite was done. John Lasseter stepped in as director and, through Herculean efforts, the film was completed on time.
- adl 5y agoThat link provides more info about the story, I remembered reading about it years ago, Also after they restored most of the files deleted, most of the movie was scraped and redone.
- frosted-flakes 5y agoScraped or scrapped? Can you explain more?
- orenjacob 5y agoscrapped - after we got this version of the film back, we decided to rewrite the story. that meant that we substantially rebuild the film in the next year, even after we had recovered the version from Galyn's machine.
- paulpauper 5y agobackup your backups
- dspillett 5y agoHaving multiple backups, on unconnected systems, is the general rule. Whether this is done in a chain (backing up the backups) or as multiple transfers from the sources is an implementation decision: ideally the latter as otherwise a break in the first backup system can stop the second updating, but this may require notably more resources at the source end so compete for those resources with production systems so the former isn't always a worse plan. More importantly: test your backups. Preferably in an automated manner to increase the change the tests are not skipped, but have humans responsible for checking the results. Ideally by doing a full restore onto alternate kit, but this is often not really practical. A backup of a backup is no extra help if the first backup is broken in a way you have not yet detected.
- closetohome 5y agoTwo is one, one is none.
- monocasa 5y ago3 - 2 - 1 3 backups 2 media formats 1 off-site
- lioeters 5y agoThe 3-2-1 Backup Rule - https://www.veeam.com/blog/321-backup-rule.html https://www.veeam.com/blog/321-backup-rule.html
- CobrastanJorji 5y agoMore importantly: test that you can restore from backup. If you're not confident that you can restore from backup, you don't have a backup.
- diego_moita 5y agoJohn Catmull (correction: sorry, his name is Ed Catmull, as tbab corrected me), Pixar co-founder and president wrote a book ("Creativity Inc") where he tells that story in detail. He wrote that everybody in the team had forgotten about Sussman's copy and she is the one that told in a meeting the copy existed, to the astonishment of everyone else. The van that went to pick up the computer on Sussman's house was equipped with pillows because they were scared even from the road's vibrations.
- Melatonic 5y agoI would have just brought some servers and storages to her house and copied it over there. Why move the machine at all? edit: Now Im really curious what this version was even like considering it was never released
- Keyframe 5y agoAnd risk waking up the baby?!
- tbabb 5y agoEd Catmull.
- deleted 5y ago[deleted]
- diego_moita 5y agoYou are right, I was wrong. Thank you.
- coldpie 5y agoWould you recommend the book?
- diego_moita 5y agoDepends on the subject of your interests. Roughly speaking the book validates the whole idea of "to build awesome stuff, hire the most creative and talented people you can find and create an environment for their imagination to run free". If you're interested in this perspective, I do very much recommend. But, obviously, he doesn't address issues like John Lasseter running free with sexual harassment and other uncomfortable issues.
- lanius 5y agoHow common was working from home in 1995?
- bityard 5y agoUnless you were an independent entity that could do most of your work over the phone with only occasional in-person meetings (e.g. certain kinds of salespeople and some artists), it was very rare. It only happened in this instance because the person in question had a newborn baby and I guess maternal leave wasn't a thing at Pixar back then.
- jedberg 5y ago> and I guess maternal leave wasn't a thing at Pixar back then. The opposite. After she finished mat leave she was allowed the special exemption to work from home so she could still spend time with her baby.
- jedberg 5y agoIt was 1998, and it wasn't that common, but in tech it wasn't impossible. For Pixar it was hard because of the size of the files, but otherwise wasn't all that hard. I occasionally worked from home in 1998. The biggest impediment was network speed -- at work we had 10mbps and sometimes even 100mpbs. But at home all we had was dail up or if you were really lucky, DSL, which was 0.384mbps, so 30 times slower, and also latency was a lot higher. So you were very limited in what you could do. Mostly it was just email and remote access ssh with tmux, where you sometimes had to deal with really high latency.
- gxqoz 5y agoThe way I heard this story is that the movie being saved wasn't actually the theatrical Toy Story 2. Instead, it was Disney's version that was mostly scrapped when they resolved their dispute with Pixar.
- rootbear 5y agoThere never was a "Disney Version" of TS2. The film that was almost lost was to be the theatrical version and it was done entirely at Pixar.
- gxqoz 5y agoYeah I think I'm mixing up my stories here. There was a "Disney Version" of Toy Story 3. And there was a very different Pixar version of Toy Story 2 that this story applies to.
- vecter 5y agoYou seem right based on Oren's Quora answer: https://news.ycombinator.com/item?id=30339069 https://news.ycombinator.com/item?id=30339069 And then, some months later, Pixar rewrote the film from almost the ground up, and we made Toy Story 2 again. That rewritten film was the one you saw in theatres and that you can watch now on Blu-Ray.
- commandlinefan 5y agoI've read that story a few times and I'm always struck by how aware management was when that did happen. I suspect that, at least half of the places I've worked, if I (or anybody else) happened to be that one employee who had a backup on their home computer, they'd find it impossible to communicate that to upper management in any way. The whole movie would be shut down while somebody was saying "I have a copy over here! I have a copy over here!"
- anyfoo 5y agoWere you actually in the situation? Because at practically all the places that I worked for (some of them smaller, some of them very big), I don't think this would be a problem if so much was at stake...
- Kaibeezy 5y agoThe same genius manager who failed to hire someone to make backups must have gone on to run the graphic design team for Avatar.
- signal11 5y agoThis story is oddly similar to the Maersk cyber-attack in 2017[1]: > The virus, once activated, propagated within just seven minutes. And for Maersk, as with other organizations, the level of destruction was enormous. > As well as affecting print and file-sharing capabilities, laptops, apps and servers, NotPetya severely damaged both Maersk’s implementations of DHCP (Dynamic Host Configuration Protocol) and Active Directory, which, respectively, allow devices to participate in networks by allocating IP addresses and provide a directory lookup for valid addresses. Along with those, the technology controlling its access to cloud services was also damaged and became unstable while its service bus was completely lost. > Banks is candid about the breadth of the impact: “There was 100% destruction of anything based on Microsoft that was attached to the network.” > In a stroke of luck, they were able to retrieve an undamaged copy of its Active Directory from the Maersk office in Nigeria. It had been unaffected by NotPetya purely because of a power outage in Lagos that had taken the service offline while the virus was spreading. A rather nervous local Maersk employee was tasked with transporting the crucial data to Denmark. What this says to me is that multiple offline backups (or failing that, copies) will save your bacon some day. [1] https://www.i-cio.com/management/insight/item/maersk-springing-back-from-a-catastrophic-cyber-attack https://www.i-cio.com/management/insight/item/maersk-springi...
- hsbauauvhabzb 5y agoOffline backups work ~now~ but it won’t be long until rootkits start corrupting data being pushed/pulled by backup servers.
- rectang 5y agoContinuously restoring from a copy of the data you sent to cold storage provides an additional layer of protection. If an attacker get root they can theoretically disable the continuous restoration, but the more moving parts they have to tweak the more likely that they'll be noticed while the org still has access to decent backups.
- hsbauauvhabzb 5y ago
- high_pathetic 5y ago
- tagoregrtst 5y agoLol, agreed, but common! It was cute.
- notorandit 5y agoA mom saved Ts2!
- fabiancook 5y agoTLDR birth causing WFM, producing a verified backup chain.
- irrational 5y ago10% of the movie was lost forever? Well, crap. Now I’m sad for the day.
- philistine 5y ago100% of the movie they were working on ended up lost. The team working on the film was not Pixar, but another studio assigned the job while Pixar were holding up during negotiations. Once those negotiations were over, Pixar started work on a sequel, and this version was never released.
- rootbear 5y agoThat is not true. I was there. The film that was nearly lost was being made at Pixar. There was never any version of TS2 made by anyone else. I think this idea may come from the fact that Disney owned the Toy Story characters and if Pixar had later gone with another partner studio, it would be Disney making Toy Story movies, not Pixar. This was a major motivator for Pixar to stay with Disney.
- orenjacob 5y agoI think the other post above is referring to the version of a Toy Story sequel that was in production down in Burbank at Disney in the early 2000's (after TS2 came out). That production was shut down when Disney and Pixar merged in the mid 2000's.
- philistine 5y agoYeah, I severely messed up and crossed my wires. Sorry!
- rootbear 5y agoAh, okay. I probably never knew about it, that was after I left Pixar. (This is Jim W. from Studio Tools, by the way. Great to see you here on HN.)
- 5y ago