14 ms·

The biggest difference, I think, was leaving the hunting for a head for a second moment, or even not doing it at all. Commitment would be very different if peo

The biggest difference, I think, was leaving the hunting for a head for a second moment, or even not doing it at all. Commitment would be very different if people were being asked to help while some heads were rolling. Because you're a real team when everybody is going in the same direction. Any call on "people, work hard do recover while we're after the moron who deleted everything" wouldn't have done it. You just commit to something when you know that you won't be under the fire if you do something wrong without knowing it.

mr_luc 10y ago
I remember this from the Field Guide to Understanding Human Error. Making recovering from human error a well-understood process is important, and as you point out, that process will work best if people aren't distracted by butt-covering.
- ChristianGeek 10y ago
  Link for those interested: https://www.amazon.com/dp/1472439058 https://www.amazon.com/dp/1472439058
cookiecaper 10y ago
Ed Catmull discusses this incident thoroughly in Creativity Inc.. He believed seeking retribution for this incident would've been counterproductive and discouraged Pixar's overall ethos as a safe place to experiment and make mistakes. It is this ethos and culture of vociferous, thorough experimentation and casting everyone's performance in the light of "What can we learn from that?" rather than "What ROI did we get from the last 3 months?" that Catmull credits for Pixar's success (paraphrasing here, but I believe this is an accurate summary). Since Catmull has an engineering background (his PhD involved the invention of the Z-buffer, and he was doing computer graphics before anyone knew anything about it), he understands that mistakes and failed projects, when combined with an forthright and collaborative feedback loop, are not problematic detours, but rather necessary mile markers on the path to real innovation. We'd be so much further ahead if we put more men like Catmull in charge of things. The biggest problem with reading Creativity Inc. is that it will rekindle the hope that there may be a sane workplace out there somewhere, when practically speaking, we know that few of us will ever find employment in one. It gave me a number of disquieting feelings as I read that the attributes of a workplace that all good engineers crave actually can and sometimes do exist out there. I had convinced myself that these things were myths, so now I'm sad that my boss isn't Ed Catmull. That said, I do believe some evaluation and/or discipline would've been appropriate in this case, not for the person who accidentally executed a command in the wrong directory, but for the people who were supposed to be maintaining backups and data integrity. Assuming that your primary job duties involve data integrity and system uptime, having non-functional backups of truly critical data stretches beyond the scope of "mistakes" and into the scope of incompetence. It is, I'm sure, very possible that no one was really assigned this task at Pixar and that it would therefore by improper to punish anyone in particular for the failure to execute it, but I do believe there is a limit between mistakes en route to innovation and negligence. My experience has been that most companies strongly take one tack or the other: they either let heads roll for minor infractions (and thus never allow good people to get established and comfortable), or they never fire anyone and let the dead weight and political fiefdoms gum up the works until the gears stop altogether. There needs to be a balance, and that's a very hard thing to find out there.
- zeveb 10y ago
  > It is, I'm sure, very possible that no one was really assigned this task at Pixar and that it would therefore by improper to punish anyone in particular for the failure to execute it, but I do believe there is a limit between mistakes en route to innovation and negligence. If indeed there was no-one assigned this task, then it was a mistake of negligence on the part of Pixar's management at the time. I'm not saying that to be snippy — that is exactly the job of management: to build the systems and processes required for employees to achieve the firm's goals. Proper backup and restore of data is one of those processes.
  - cookiecaper 10y ago
    Yeah, I understand that, but at the same time, backups and security, while being among the two most critical aspects of IT and computer infrastructure at a company, are often the most overlooked by everyone. That persists today and I'm sure it was even more the case back then. If management can't give it the time of day, how can an employee be expected to do so? An executive usually requires a "Come to Jesus" moment like this one, where the entire company teeters on the precipice due to lax backup or security policies, to really have the importance impressed upon him or her. At that point, they are generally much more supportive, though sadly, this too can start to fade if the sysadmins do their job too well. I don't want anyone to come away thinking that most companies have solved these problems. It's definitely not the case, even in large, established companies. Security and backups continue to get little attention until it's too late. We really need to call a celebrity in MBA circles and get that person to run a seminar meant to scare the pants off the execs.
    - greedo 10y ago
      We had a proper backup system in place at my company. Backups were replicated to an identical system at a remote site, and we periodically validated random backups. Our audit team also spot checked our backups to ensure that all servers were being backed up. Then we had a management change; ditched our system for a more expensive solution, and basically told our application owners that backups were a nice thing to have, but they shouldn't really count on them. The person in charge of backups was assigned other tasks and encouraged not to spend too much time on managing the backup system. With over 1000 servers, it's a matter of time until we have an issue that leads to a CTJ event. Unfortunately, the backup admin will probably be the one facing Jesus, not the management staff...
FabHK 10y ago
Very good point. Aviation is awesome in that sense - accident investigations are focused on understanding what happened, and preventing re-occurence. Allocating blame or punishment are not part of it, at least in enlightened jurisdictions. Furthermore, a lot of individual error's are seen in an institutionalised "systems" framework - given that people invariably will make mistakes, how can we set up the environment/institutions/systems so that errors are not catastrophic. Not sure how that applies to movie animation, to be honest, but not primarily looking for whom to blame was certainly a very good move.
- dweekly 10y ago
  Rotorcraft PPL here: actually assigning blame is a very important part of accident analysis because it's critical to determining whether resolution will involve modifying training curriculum or if there was a mechanical failure. CFIT, engine failure due to fuel exhaustion, flight into wires, low G mast bumping, and settling with power are all sure ways to die from pilot error. And if the pilot did make a serious error their license could be suspended or revoked.
  - ColinCochrane 10y ago
    This is true, but I think the point the parent comment was getting at is that an investigation tends to take a more holistic look at the incident rather than simply assigning blame directly to a single factor. Even pilot error, especially in the context of commercial aviation, is often found to be the result of training deficiencies or cultural issues on the part of the airline.
    - FabHK 10y ago
      Yes, exactly, well said. That's precisely the idea: if a pilot makes errors so grave as to endanger the aircraft, how come the airline training/monitoring in place did not pick up indications earlier? The people that helped overcome the "pilot error, case closed"-mindset must be thanked (among others) for making aviation today as safe as it is.
  - 10y ago
lokedhs 10y ago
I never understood the attitude of some companies to fire an employee immediately if they make a mistake such as accidentally deleting some files. If you keep this employee, then you can e pretty sure he'll never made that mistake again. If you fire him and hire someone else, that person might not have had the learning experience of completely screwing up a system. I think that employees actually makes less mistakes and are more productive if they don't have be worried about being fired for making a mistake.
- gech 10y ago
  Just like how a new executive is brought into a failing company, the company still fails but the executive is awarded a nice severance package.
  - SEJeff 10y ago
    Sounds like the life story of Yahoo.
- knicholes 10y ago
  "Why would I fire you? I just paid $10M to train you to not that make mistake again!"
  - modfodder 10y ago
    I had a great boss (founder of the company) who said, after I just screwed up, "There is not a mistake you can make, that I haven't already made. Just don't make the same mistake twice."
    - dingaling 10y ago
      Reminds me of the boot-camp story of the nervous recruit inquiring of his sergeant: "Sir beg my pardon for asking but why did you give Smith 50 press-ups for asking a question? You said that there were no stupid questions. Sir." "I gave him the press-ups the SECOND time he asked. Will you need to ask again?"
weavie 10y ago
The fault here does not lie with just one person. One person ran the rm -rf command. Other people failed to check the backups. Others made the decision to give everyone full root access. Really it was a large part of the company that was to blame.
- StavrosK 10y ago
  Whenever there's a bug in code I reviewed, there are at least two people responsible: The person who wrote the code and me, the person who reviewed it. I've found that that helps morale, as there's a sense of shared responsibility, but there's no blaming people for problems where I work, so I haven't actually seen what happens when people are searching for the culprit. The usual process is "this happened because of this, this and this all went wrong to cause us to not notice the problem, and we've added this to make it less likely that it will happen again". If you have smart, experienced people and you get problems, it's because of your processes, not the people, so the former is what you should fix.
  - mulmen 10y ago
    Everywhere I have worked took the shared responsibility approach, I think that's the status quo but there are obviously exceptions. One day at $megacorp a bug caused a production outage and a project manager sent an email to the team calling out the engineer who committed the change and asking us all to be more careful in the future. That manager was immediately reprimanded both in the email chain and in private. I find the culture of shared responsibility to be one of the best qualities of our industry, even if it isn't universal.
- orbitingpluto 10y ago
  Oh god. I know someone who made all of these mistakes by themselves in a certain week. And kept his job. He was pretty. I left and found out two months later from a friend he had managed to take down almost every single server in the place for which he had access. Even the legacy don't touch systems that just boot and run equipment. Be pretty.
gist 10y ago
> You just commit to something when you know that you won't be under the fire if you do something wrong without knowing it. The other side is if you play a key role (and head could roll after the hard work is done) to simply leverage that fact (perhaps with others) as an advantage such that you have get a new contract and can't be fired for X amount of time.
tbabb 10y ago
I absolutely agree with this. There was an incident where I work where an employee (a new hire) set up a cron job to delete his local code tree, re-sync a new copy, then re-build it using a cron job every night. A completely reasonable thing for a coder to automate. In his crontab he put: rm -rf /$MY_TREE_ROOT and as everyone undoubtedly first discovered by accident, the crontab environment is stripped bare of all your ordinary shell environment. So $MY_TREE_ROOT expanded to "". The crontab ran on Friday, IIRC, and got most of the way through deleting the entire project over the weekend before a lead came in and noticed things were disappearing. Work was more or less halted for several days while the supes worked to restore everything from backup. Could the blunder have been prevented? Yes, probably with a higher degree of caution, but that level of subtlety in a coding mistake is made regularly by most people (especially someone right out of university); he was just unlucky that the consequences were catastrophic, and that he tripped over the weird way crontab works in the worst possible usage case. He probably even tested it in his shell. We all know to quadruple-check our rm-rfs, but we know that because we've all made (smaller) mistakes like his. It could have been anyone. Dragging him to the guillotine would have solved nothing. In fact, the real question is "how is it possible for such an easy mistake to hose the entire project?" Some relatively small permissions changes at the directory root of the master project probably would have stopped stray `rm -rf`s from escaping local machines without severely sacrificing the frictionless environment that comes from an open-permissions shop. So if anything, the failure was systems's fault for setting up a filesystem that can be hosed so easily and so completely by an honest mistake. The correct thing to do was (and is) to skip the witch hunt, and focus on problem-solving. I am not sure, but I think the employee was eventually hired on at the end of his internship. For me the principle is: Standards and habits are teachable. Competence and attitude, less so. Educate and train for the former, and a failure of the former should cause you to look first at your procedures, not the people. Hire and fire for the latter.

mr_luc 10y ago

I remember this from the Field Guide to Understanding Human Error. Making recovering from human error a well-understood process is important, and as you point out, that process will work best if people aren't distracted by butt-covering.

ChristianGeek 10y ago
Link for those interested: https://www.amazon.com/dp/1472439058 https://www.amazon.com/dp/1472439058

cookiecaper 10y ago

Ed Catmull discusses this incident thoroughly in Creativity Inc.. He believed seeking retribution for this incident would've been counterproductive and discouraged Pixar's overall ethos as a safe place to experiment and make mistakes. It is this ethos and culture of vociferous, thorough experimentation and casting everyone's performance in the light of "What can we learn from that?" rather than "What ROI did we get from the last 3 months?" that Catmull credits for Pixar's success (paraphrasing here, but I believe this is an accurate summary). Since Catmull has an engineering background (his PhD involved the invention of the Z-buffer, and he was doing computer graphics before anyone knew anything about it), he understands that mistakes and failed projects, when combined with an forthright and collaborative feedback loop, are not problematic detours, but rather necessary mile markers on the path to real innovation. We'd be so much further ahead if we put more men like Catmull in charge of things. The biggest problem with reading Creativity Inc. is that it will rekindle the hope that there may be a sane workplace out there somewhere, when practically speaking, we know that few of us will ever find employment in one. It gave me a number of disquieting feelings as I read that the attributes of a workplace that all good engineers crave actually can and sometimes do exist out there. I had convinced myself that these things were myths, so now I'm sad that my boss isn't Ed Catmull. That said, I do believe some evaluation and/or discipline would've been appropriate in this case, not for the person who accidentally executed a command in the wrong directory, but for the people who were supposed to be maintaining backups and data integrity. Assuming that your primary job duties involve data integrity and system uptime, having non-functional backups of truly critical data stretches beyond the scope of "mistakes" and into the scope of incompetence. It is, I'm sure, very possible that no one was really assigned this task at Pixar and that it would therefore by improper to punish anyone in particular for the failure to execute it, but I do believe there is a limit between mistakes en route to innovation and negligence. My experience has been that most companies strongly take one tack or the other: they either let heads roll for minor infractions (and thus never allow good people to get established and comfortable), or they never fire anyone and let the dead weight and political fiefdoms gum up the works until the gears stop altogether. There needs to be a balance, and that's a very hard thing to find out there.

zeveb 10y ago
> It is, I'm sure, very possible that no one was really assigned this task at Pixar and that it would therefore by improper to punish anyone in particular for the failure to execute it, but I do believe there is a limit between mistakes en route to innovation and negligence. If indeed there was no-one assigned this task, then it was a mistake of negligence on the part of Pixar's management at the time. I'm not saying that to be snippy — that is exactly the job of management: to build the systems and processes required for employees to achieve the firm's goals. Proper backup and restore of data is one of those processes.
- cookiecaper 10y ago
  Yeah, I understand that, but at the same time, backups and security, while being among the two most critical aspects of IT and computer infrastructure at a company, are often the most overlooked by everyone. That persists today and I'm sure it was even more the case back then. If management can't give it the time of day, how can an employee be expected to do so? An executive usually requires a "Come to Jesus" moment like this one, where the entire company teeters on the precipice due to lax backup or security policies, to really have the importance impressed upon him or her. At that point, they are generally much more supportive, though sadly, this too can start to fade if the sysadmins do their job too well. I don't want anyone to come away thinking that most companies have solved these problems. It's definitely not the case, even in large, established companies. Security and backups continue to get little attention until it's too late. We really need to call a celebrity in MBA circles and get that person to run a seminar meant to scare the pants off the execs.
  - greedo 10y ago
    We had a proper backup system in place at my company. Backups were replicated to an identical system at a remote site, and we periodically validated random backups. Our audit team also spot checked our backups to ensure that all servers were being backed up. Then we had a management change; ditched our system for a more expensive solution, and basically told our application owners that backups were a nice thing to have, but they shouldn't really count on them. The person in charge of backups was assigned other tasks and encouraged not to spend too much time on managing the backup system. With over 1000 servers, it's a matter of time until we have an issue that leads to a CTJ event. Unfortunately, the backup admin will probably be the one facing Jesus, not the management staff...

FabHK 10y ago

Very good point. Aviation is awesome in that sense - accident investigations are focused on understanding what happened, and preventing re-occurence. Allocating blame or punishment are not part of it, at least in enlightened jurisdictions. Furthermore, a lot of individual error's are seen in an institutionalised "systems" framework - given that people invariably will make mistakes, how can we set up the environment/institutions/systems so that errors are not catastrophic. Not sure how that applies to movie animation, to be honest, but not primarily looking for whom to blame was certainly a very good move.

dweekly 10y ago
Rotorcraft PPL here: actually assigning blame is a very important part of accident analysis because it's critical to determining whether resolution will involve modifying training curriculum or if there was a mechanical failure. CFIT, engine failure due to fuel exhaustion, flight into wires, low G mast bumping, and settling with power are all sure ways to die from pilot error. And if the pilot did make a serious error their license could be suspended or revoked.
- ColinCochrane 10y ago
  This is true, but I think the point the parent comment was getting at is that an investigation tends to take a more holistic look at the incident rather than simply assigning blame directly to a single factor. Even pilot error, especially in the context of commercial aviation, is often found to be the result of training deficiencies or cultural issues on the part of the airline.
  - FabHK 10y ago
    Yes, exactly, well said. That's precisely the idea: if a pilot makes errors so grave as to endanger the aircraft, how come the airline training/monitoring in place did not pick up indications earlier? The people that helped overcome the "pilot error, case closed"-mindset must be thanked (among others) for making aviation today as safe as it is.
- 10y ago

lokedhs 10y ago

I never understood the attitude of some companies to fire an employee immediately if they make a mistake such as accidentally deleting some files. If you keep this employee, then you can e pretty sure he'll never made that mistake again. If you fire him and hire someone else, that person might not have had the learning experience of completely screwing up a system. I think that employees actually makes less mistakes and are more productive if they don't have be worried about being fired for making a mistake.

gech 10y ago
Just like how a new executive is brought into a failing company, the company still fails but the executive is awarded a nice severance package.
- SEJeff 10y ago
  Sounds like the life story of Yahoo.
knicholes 10y ago
"Why would I fire you? I just paid $10M to train you to not that make mistake again!"
- modfodder 10y ago
  I had a great boss (founder of the company) who said, after I just screwed up, "There is not a mistake you can make, that I haven't already made. Just don't make the same mistake twice."
  - dingaling 10y ago
    Reminds me of the boot-camp story of the nervous recruit inquiring of his sergeant: "Sir beg my pardon for asking but why did you give Smith 50 press-ups for asking a question? You said that there were no stupid questions. Sir." "I gave him the press-ups the SECOND time he asked. Will you need to ask again?"

weavie 10y ago

The fault here does not lie with just one person. One person ran the rm -rf command. Other people failed to check the backups. Others made the decision to give everyone full root access. Really it was a large part of the company that was to blame.

StavrosK 10y ago
Whenever there's a bug in code I reviewed, there are at least two people responsible: The person who wrote the code and me, the person who reviewed it. I've found that that helps morale, as there's a sense of shared responsibility, but there's no blaming people for problems where I work, so I haven't actually seen what happens when people are searching for the culprit. The usual process is "this happened because of this, this and this all went wrong to cause us to not notice the problem, and we've added this to make it less likely that it will happen again". If you have smart, experienced people and you get problems, it's because of your processes, not the people, so the former is what you should fix.
- mulmen 10y ago
  Everywhere I have worked took the shared responsibility approach, I think that's the status quo but there are obviously exceptions. One day at $megacorp a bug caused a production outage and a project manager sent an email to the team calling out the engineer who committed the change and asking us all to be more careful in the future. That manager was immediately reprimanded both in the email chain and in private. I find the culture of shared responsibility to be one of the best qualities of our industry, even if it isn't universal.
orbitingpluto 10y ago
Oh god. I know someone who made all of these mistakes by themselves in a certain week. And kept his job. He was pretty. I left and found out two months later from a friend he had managed to take down almost every single server in the place for which he had access. Even the legacy don't touch systems that just boot and run equipment. Be pretty.

gist 10y ago

> You just commit to something when you know that you won't be under the fire if you do something wrong without knowing it. The other side is if you play a key role (and head could roll after the hard work is done) to simply leverage that fact (perhaps with others) as an advantage such that you have get a new contract and can't be fired for X amount of time.

tbabb 10y ago

I absolutely agree with this. There was an incident where I work where an employee (a new hire) set up a cron job to delete his local code tree, re-sync a new copy, then re-build it using a cron job every night. A completely reasonable thing for a coder to automate. In his crontab he put: rm -rf /$MY_TREE_ROOT and as everyone undoubtedly first discovered by accident, the crontab environment is stripped bare of all your ordinary shell environment. So $MY_TREE_ROOT expanded to "". The crontab ran on Friday, IIRC, and got most of the way through deleting the entire project over the weekend before a lead came in and noticed things were disappearing. Work was more or less halted for several days while the supes worked to restore everything from backup. Could the blunder have been prevented? Yes, probably with a higher degree of caution, but that level of subtlety in a coding mistake is made regularly by most people (especially someone right out of university); he was just unlucky that the consequences were catastrophic, and that he tripped over the weird way crontab works in the worst possible usage case. He probably even tested it in his shell. We all know to quadruple-check our rm-rfs, but we know that because we've all made (smaller) mistakes like his. It could have been anyone. Dragging him to the guillotine would have solved nothing. In fact, the real question is "how is it possible for such an easy mistake to hose the entire project?" Some relatively small permissions changes at the directory root of the master project probably would have stopped stray `rm -rf`s from escaping local machines without severely sacrificing the frictionless environment that comes from an open-permissions shop. So if anything, the failure was systems's fault for setting up a filesystem that can be hosed so easily and so completely by an honest mistake. The correct thing to do was (and is) to skip the witch hunt, and focus on problem-solving. I am not sure, but I think the employee was eventually hired on at the end of his internship. For me the principle is: Standards and habits are teachable. Competence and attitude, less so. Educate and train for the former, and a failure of the former should cause you to look first at your procedures, not the people. Hire and fire for the latter.