5 ms·

Ed Catmull discusses this incident thoroughly in Creativity Inc.. He believed seeking retribution for this incident would've been counterproductive and discoura

Ed Catmull discusses this incident thoroughly in Creativity Inc.. He believed seeking retribution for this incident would've been counterproductive and discouraged Pixar's overall ethos as a safe place to experiment and make mistakes. It is this ethos and culture of vociferous, thorough experimentation and casting everyone's performance in the light of "What can we learn from that?" rather than "What ROI did we get from the last 3 months?" that Catmull credits for Pixar's success (paraphrasing here, but I believe this is an accurate summary). Since Catmull has an engineering background (his PhD involved the invention of the Z-buffer, and he was doing computer graphics before anyone knew anything about it), he understands that mistakes and failed projects, when combined with an forthright and collaborative feedback loop, are not problematic detours, but rather necessary mile markers on the path to real innovation. We'd be so much further ahead if we put more men like Catmull in charge of things. The biggest problem with reading Creativity Inc. is that it will rekindle the hope that there may be a sane workplace out there somewhere, when practically speaking, we know that few of us will ever find employment in one. It gave me a number of disquieting feelings as I read that the attributes of a workplace that all good engineers crave actually can and sometimes do exist out there. I had convinced myself that these things were myths, so now I'm sad that my boss isn't Ed Catmull. That said, I do believe some evaluation and/or discipline would've been appropriate in this case, not for the person who accidentally executed a command in the wrong directory, but for the people who were supposed to be maintaining backups and data integrity. Assuming that your primary job duties involve data integrity and system uptime, having non-functional backups of truly critical data stretches beyond the scope of "mistakes" and into the scope of incompetence. It is, I'm sure, very possible that no one was really assigned this task at Pixar and that it would therefore by improper to punish anyone in particular for the failure to execute it, but I do believe there is a limit between mistakes en route to innovation and negligence. My experience has been that most companies strongly take one tack or the other: they either let heads roll for minor infractions (and thus never allow good people to get established and comfortable), or they never fire anyone and let the dead weight and political fiefdoms gum up the works until the gears stop altogether. There needs to be a balance, and that's a very hard thing to find out there.

zeveb 10y ago
> It is, I'm sure, very possible that no one was really assigned this task at Pixar and that it would therefore by improper to punish anyone in particular for the failure to execute it, but I do believe there is a limit between mistakes en route to innovation and negligence. If indeed there was no-one assigned this task, then it was a mistake of negligence on the part of Pixar's management at the time. I'm not saying that to be snippy — that is exactly the job of management: to build the systems and processes required for employees to achieve the firm's goals. Proper backup and restore of data is one of those processes.
- cookiecaper 10y ago
  Yeah, I understand that, but at the same time, backups and security, while being among the two most critical aspects of IT and computer infrastructure at a company, are often the most overlooked by everyone. That persists today and I'm sure it was even more the case back then. If management can't give it the time of day, how can an employee be expected to do so? An executive usually requires a "Come to Jesus" moment like this one, where the entire company teeters on the precipice due to lax backup or security policies, to really have the importance impressed upon him or her. At that point, they are generally much more supportive, though sadly, this too can start to fade if the sysadmins do their job too well. I don't want anyone to come away thinking that most companies have solved these problems. It's definitely not the case, even in large, established companies. Security and backups continue to get little attention until it's too late. We really need to call a celebrity in MBA circles and get that person to run a seminar meant to scare the pants off the execs.
  - greedo 10y ago
    We had a proper backup system in place at my company. Backups were replicated to an identical system at a remote site, and we periodically validated random backups. Our audit team also spot checked our backups to ensure that all servers were being backed up. Then we had a management change; ditched our system for a more expensive solution, and basically told our application owners that backups were a nice thing to have, but they shouldn't really count on them. The person in charge of backups was assigned other tasks and encouraged not to spend too much time on managing the backup system. With over 1000 servers, it's a matter of time until we have an issue that leads to a CTJ event. Unfortunately, the backup admin will probably be the one facing Jesus, not the management staff...

zeveb 10y ago

> It is, I'm sure, very possible that no one was really assigned this task at Pixar and that it would therefore by improper to punish anyone in particular for the failure to execute it, but I do believe there is a limit between mistakes en route to innovation and negligence. If indeed there was no-one assigned this task, then it was a mistake of negligence on the part of Pixar's management at the time. I'm not saying that to be snippy — that is exactly the job of management: to build the systems and processes required for employees to achieve the firm's goals. Proper backup and restore of data is one of those processes.

cookiecaper 10y ago
Yeah, I understand that, but at the same time, backups and security, while being among the two most critical aspects of IT and computer infrastructure at a company, are often the most overlooked by everyone. That persists today and I'm sure it was even more the case back then. If management can't give it the time of day, how can an employee be expected to do so? An executive usually requires a "Come to Jesus" moment like this one, where the entire company teeters on the precipice due to lax backup or security policies, to really have the importance impressed upon him or her. At that point, they are generally much more supportive, though sadly, this too can start to fade if the sysadmins do their job too well. I don't want anyone to come away thinking that most companies have solved these problems. It's definitely not the case, even in large, established companies. Security and backups continue to get little attention until it's too late. We really need to call a celebrity in MBA circles and get that person to run a seminar meant to scare the pants off the execs.
- greedo 10y ago
  We had a proper backup system in place at my company. Backups were replicated to an identical system at a remote site, and we periodically validated random backups. Our audit team also spot checked our backups to ensure that all servers were being backed up. Then we had a management change; ditched our system for a more expensive solution, and basically told our application owners that backups were a nice thing to have, but they shouldn't really count on them. The person in charge of backups was assigned other tasks and encouraged not to spend too much time on managing the backup system. With over 1000 servers, it's a matter of time until we have an issue that leads to a CTJ event. Unfortunately, the backup admin will probably be the one facing Jesus, not the management staff...