8 ms·
Pixar almost deleted Toy Story 2 (2012)
- deleted 4y ago[deleted]
- deleted 4y ago[deleted]
- dingosity 4y agoThis is a great example of why it's a good idea to always test your backups.
- ltbarcly3 4y agoI came to make this snide comment as well! Nobody tests their backups though, I've only ever seen it done inadvertently when the backups were used as part of the development process.
- yakubin 4y agoBTW how do you test a personal workstation backup? I don’t have backup workstations to rotate between by backing one up, restoring on the other, working and then cycling between them in this manner. How do you test the restore procedure without risking destruction of the original?
- opencl 4y agoYou can buy an extra storage device and swap them out (either physically or swapping the boot order), as long as your workstation doesn't have soldered in storage.
- jen20 4y agoEven if it does, you can usually boot from an external volume anyway.
- l33tman 4y agoMost backup systems allow you to explore backups and extract individual files. Testing a bunch of files for restoring is at least better than completely ignoring it.. however it's difficult if it turns out part of your backup is simply missing, maybe an erroneous exclusion entry or some path miss (happened to me, I was without backup for a home folder for a year before I noticed it, the /home/user/ was a symlink to another volume that mostly had tmp data and so wasn't backed up). You might not even notice this even if you restore the whole backup, because you can't really be expected to go through everything every time either..
- b112 4y agomd5sum using a bash script and find, then compare on live.
- vikingerik 4y agoYou don't need backups. You need restores.
- pstuart 4y agoIn the olden days when I was the admin for a small shop we would immediately restore the backup to a second system to be used as a playground. Win/win: verified restore and a place to test/train with production data.
- sixothree 4y agoYup, just as long as none of the connection strings point to (and can reach) a prod database.
- brainwipe 4y agoIt's 2022. Test your backups.
- haswell 4y agoWhile yes, backups should be tested, this all happened over two decades ago.
- DefNotMe 4y agoIs kottke.org operated by Daniel Kottke?
- jonhohle 4y agoJason Kottke, unrelated - https://kottke.org/99/06/dan-kottke https://kottke.org/99/06/dan-kottke
- jefftk 4y agoMore details on Quora: https://www.quora.com/Pixar-company/Did-Pixar-accidentally-delete-Toy-Story-2-during-production/answer/Oren-Jacob https://www.quora.com/Pixar-company/Did-Pixar-accidentally-d... The ending is sad -- that intrepid recovery mostly didn't end up used: And then, some months later, Pixar rewrote the film from almost the ground up, and we made Toy Story 2 again. That rewritten film was the one you saw in theatres and that you can watch now on Blu-Ray.
- haswell 4y agoI don't know about sad necessarily - if the earlier iteration was necessary to realize they needed to make the changes that brought us the final version, it's all part of a larger journey. I can see how this could be frustrating for people involved, but reshoots / rewrites are part of the process. And I suspect that the recovery of character models and other assets was still instrumental in making the final film, if not the actual shots/sequences themselves.
- trynewideas 4y agoConsidering how good Toy Story 2 was, "sad" is an odd way of putting it, unless you're talking about the ensuing crunch. There are a few more details about that side of the story elsewhere: https://www.independent.co.uk/arts-entertainment/films/news/lightyear-toy-story-2-deleted-b2017238.html https://www.independent.co.uk/arts-entertainment/films/news/... https://www.denofgeek.com/movies/toy-story-2-disasters-almost-killed-movie/ https://www.denofgeek.com/movies/toy-story-2-disasters-almos...
- nathanvanfleet 4y agoThe sad part is that the thing they saved didn't actually get used and was just thrown out. So it wasn't exactly the big save it's made up to be. But if it wasn't framed as a big save maybe it wouldn't be posted every week or so.
- haswell 4y agoIf the thing they saved hadn’t been saved, it makes me wonder if they would have remade it as written, and then either have to throw that out or release what was by all accounts not a good first pass. I suspect some character models were reused, and recovering what they did enabled the rewrite that came after. I think it’s still fair to characterize this as a big save even if it wasn’t used as-is. They arguable needed the first version to decide to move away from it.
- geocrasher 4y agoMy favorite part of this account isn't the "test your backups!" lesson. It's the "Have an amazing team" lesson. In one weekend, a team of actual humans sorted through thirty thousand files by hand. Surround yourself with amazing people, and you will find amazing solutions.
- outside1234 4y agoand then make sure those amazing people are testing their backups. :)
- bluedino 4y ago>> Galyn mentioned she might have a copy at her house. So we went home to get that machine, and you can watch the video for how that went… DLP software hadn't yet been invented
- helsinkiandrew 4y agoI think there’s more background and detail in this version: https://thenextweb.com/news/how-pixars-toy-story-2-was-deleted-twice-once-by-technology-and-again-for-its-own-good https://thenextweb.com/news/how-pixars-toy-story-2-was-delet...
- dang 4y agoWe've changed the URL to that from https://kottke.org/12/05/how-pixar-almost-deleted-toy-story-2 https://kottke.org/12/05/how-pixar-almost-deleted-toy-story-.... Thanks!
- naikrovek 4y agothis is insane to me because individual frames for a Pixar film are around 1GB in size. recall how large laptop hard drives were before Toy Story 2 (2002)... I have an extremely small amount of experience writing shaders for RenderMan (BMRT, actually), and those aren't large. animation data typically isn't HUGE, but models sure as heck are, and a frame you send to RenderMan is concrete (it used to be, maybe it isn't anymore?) meaning that all geometry was present in the file you gave to RenderMan, only the compiled shaders were referenced from it, because they were procedural and could not be statically included. so I guess the frames the renderer sees are sort of like the binaries of what I would now view as a software compilation process.
- raldi 4y agoI think this story is about the models, not the rendered frames.
- naikrovek 4y agowell that's my point. I always thought that what was lost were frame definition files, and not the models. surely every animator had copies of the models; you can't animate without them, and the movie was in production when the data was lost, so they had those models, or certainly most of them. I think what was lost was the animations, and not the models or the frame/scene files.
- raldi 4y agoI'm reminded of the time I was on the InfoSec team at a company you've all heard of where all software development was done on a handful of Unix servers with a shared network file system. I reached out to the Systems team to suggest they look into mounting the home directories with the no-setuid option, and one of them invited me over to pair at their desk. They explained that there was a trick, where you could leave the existing mount point as is, with its existing mount options, and then mount the same device somewhere else with different options. So they got a root shell, went into some scratch directory, mounted the homedirs there, and we tested things out. I forget how that went, or what happened next, but it seems that we neglected to clean up after this experiment, and at some point, another Systems engineer was looking around the scratch directory and perhaps ran something like `du -ak` (forgetting to specify the option that disables crossing onto new devices), finding what appeared to be a gigantic copy of everyone's home directories. So they started `rm -rf`'ing it, and made a pretty huge dent before the phones started ringing and they realized that what they were erasing was not a copy on a local scratch space, but rather the real thing over on the NFS drive. Fortunately, this company did have great backups.
- LanceH 4y agoI was one ROLLBACK away from the Users table in production. Mid 90's, we fixed things in production all the time, and I left the where clause out. There were no soft deletes back then, we just deleted the row, and that would trigger the deletion of user resources elsewhere... I'm sure we had backups, but it would have been bad and resulted in a lot more fixes in production. I guess the upside of the "seat of the pants" days is that there was poor referential integrity, so it wouldn't have taken out anything else like billing history. edit: I never typed anything so carefully as I typed that rollback. I almost want to say there was an autocommit when closing the session, but my memory on that is fuzzy. MS SQL Server, if anyone knows. From then on, when called on for sketchy but necessary actions, I write the where clause first, run it as a select, see the number of results, wrap it in a transaction, etc...
- tomcam 4y agoGood story although there were indeed soft deletes in the 90s.
- dang 4y agoRelated: A baby saved ‘Toy Story 2’ from near complete deletion (2021) - https://news.ycombinator.com/item?id=30330740 https://news.ycombinator.com/item?id=30330740 - Feb 2022 (121 comments) Toy Story 2 files wiped by accident, restored thanks to work-from-home employee - https://news.ycombinator.com/item?id=23022717 https://news.ycombinator.com/item?id=23022717 - April 2020 (2 comments) How Pixar’s Toy Story 2 was deleted twice (2012) - https://news.ycombinator.com/item?id=20271179 https://news.ycombinator.com/item?id=20271179 - June 2019 (2 comments) Toy Story 2 Got Deleted Twice, Once on Accident, Again on Purpose (2012) - https://news.ycombinator.com/item?id=20265328 https://news.ycombinator.com/item?id=20265328 - June 2019 (4 comments) How One Line of Text Nearly Killed Toy Story 2 - https://news.ycombinator.com/item?id=16621848 https://news.ycombinator.com/item?id=16621848 - March 2018 (1 comment) How Pixar’s Toy Story 2 was deleted twice [] (2012) - https://news.ycombinator.com/item?id=15978145 https://news.ycombinator.com/item?id=15978145 - Dec 2017 (1 comment) Did Pixar accidentally delete Toy Story 2 during production? (2012) - https://news.ycombinator.com/item?id=13417037 https://news.ycombinator.com/item?id=13417037 - Jan 2017 (253 comments) How Toy Story 2 Got Deleted Twice, Once on Accident, Again on Purpose - https://news.ycombinator.com/item?id=11500467 https://news.ycombinator.com/item?id=11500467 - April 2016 (1 comment) Rm -rf nearly killed Toy Story 2 - https://news.ycombinator.com/item?id=11326831 https://news.ycombinator.com/item?id=11326831 - March 2016 (1 comment) How Toy Story 2 was nearly lost - https://news.ycombinator.com/item?id=9739675 https://news.ycombinator.com/item?id=9739675 - June 2015 (1 comment) Pixar’s Toy Story 2 was deleted twice (2012) - https://news.ycombinator.com/item?id=8786790 https://news.ycombinator.com/item?id=8786790 - Dec 2014 (2 comments) How Pixar’s Toy Story 2 was deleted twice once by tech & again for its own good - https://news.ycombinator.com/item?id=4001714 https://news.ycombinator.com/item?id=4001714 - May 2012 (1 comment) How Pixar Almost Lost Toy Story 2 To A Bad Backup - https://news.ycombinator.com/item?id=3972798 https://news.ycombinator.com/item?id=3972798 - May 2012 (75 comments)
- onionisafruit 4y agoIt looks like you're saying you've had this conversation before.
- dang 4y ago
- jfb 4y agoI remember learning that when Pixar did a pan-and-scan version of one of their movies, they wouldn't just use a floating 4:3 frame over the 16:9 original -- they'd re-render the movie, moving items around as necessary, to get a better framing. The perfectionism was palpable when we'd meet with them.
- bee_rider 4y agoThis seems like a huge advantage with mostly-digital movies. I wonder how easily a Marvel or Starwars movie could be re-rendered like this. Technically there are actors of course, but they are usually the main focus of the scene anyway...
- gabereiser 4y agoFor green screen shots this would be fine. For VFX LED wall shots, this wouldn’t work. However, movies and shows today are shot in a high resolution that you can crop/edit into most formats. Green screen shots lend itself to resizing better due to the ability to fill in more of the scene while pulling the subject matter away a bit to fit the format.
- jasonjamerson 4y agoMany LED wall shots are completely rotoscoped anyway just to dial it in that much further. But you still get the advantages of environmental lighting and the efficiency of real time decision making.
- jonny_eh 4y agoVFX shots are not pure CGI, it's a whole lot of layers composited on top of each other, think Photoshop but video. So making tweaks to any given shot would not be as easy.
- hondo77 4y ago> I wonder how easily a Marvel or Starwars movie could be re-rendered like this. Fortunately we don't have to worry about 4:3 anymore.
- navels 4y agoDo people not search HN before posting?
- locallost 4y agoI remember the stories from Ed Catmull's book Creativity Inc. Although not that they tossed the movie out after almost deleting it anyway, like it's said in the article. The story about starting from scratch was that they left the team that did it alone, but it was their first movie and it fell flat. Reading about it then felt like a dick move and kind of questionable, but even more so later when I'd read that Lassiter, who was the most critical one, is a certified creep / jerk. So I don't know, but I guess if you're successful in the end your opinion is never questioned.
- sdevonoes 4y ago> In order for her to work on the movie while out, they had plugged the machine up to the local network and copied the whole file tree over. Now image the same situation in 2022: - there is no acceptable backup for the whole movie (or let's say, app) - only you know there is one acceptable copy... but it's on your personal laptop. You did copy it back in the day because, well, let's say you wanted to run the app in our local machine There is no way you would tell your team/boss: "Hey guys, no worries! I have a perfect copy of it!", because why on earth did you copy work material into your personal laptop?! That's the rule number 1 of every modern tech company: don't mix work stuff with personal laptop (and the other way around: don't use work laptop for personal stuff).
- asciimov 4y ago- If you can trust an employee with a $25K ($50K today) computer, trusting them with a complete source tree isn't a stretch. - I'd be more pissed off that somebody didn't have a complete backup somewhere, even if it was a personal laptop they used for work. - Back then, most people didn't have home computers. Hell nowadays a home pc isn't a guarantee, as most of my (non-engineering) coworkers only have cellphones and ipads at home. And while I agree don't mix work and play, most of these people do.
- serf 4y ago> Back then, most people didn't have home computers. you had a much different perspective of 99 than I did. That factoid might have been true for-the-public-at-large, but certainly not for the offices of large CGI or dev firms.
- asciimov 4y ago> for-the-public-at-large True. Most of the people I knew with computers back then had them provided by their work, or bought them for university. But then I lived in a small city in Texas not near a major metro. Most people there didn't buy a home pc until the DotCom boom brought those cheap emachines.
- deleted 4y ago[deleted]
- cnorthwood 4y agoEarly in my career I was working for a well-known anti-virus firm. They had a crash of their Perforce servers and it turns out the backups were not good and the most recent good one was several months old. They painstakingly restored it from the good backup and forensic recovery of the drives, but there were several files that were still missing to fully restore the archive. They did know the hashes of the files, so what they did was push out a bunch of custom virus signatures to the dogfooding (internal testing) environment which flagged any files with those hashes as viruses, and then asked anyone who had a hit to send the file into central IT to get it resolved. It was fairly clever I thought!
- fortran77 4y agoThe real culprit in this story isn't the person who may have accidentally typed an "rm -rf" -- it's the person who was responsible for making backups.
- StevePerkins 4y agoEveryone talking about backups, and the need to avoid blame during disaster recovery. And that's all great. But MY big takeaway is the final section. After all that, the studio threw away the movie anyway. Had their people make a different movie, working 100 hours per week for 9 months straight. That sounds like an absolutely dystopian hellscape of a culture, and the tone of the article seems to celebrate it. Marveling that those people had the "opportunity" to experience that level of "camaraderie". My first couple of jobs out of college were pretty "death march"-ish. And yeah, maybe there's something to having an experience or two like that under your belt when you're young and single. But this Pixar example here is just straight up abusive and exploitive, and I'm not sure that a celebratory lens is how we should best view it.
- WirelessGigabit 4y agoDoes someone have Red Alert 2 stored somewhere?