3 ms·

I think when the circumstances actually show someone was grossly negligent, it's very hard to convincingly pretend that only processes are to blame. Remember th

I think when the circumstances actually show someone was grossly negligent, it's very hard to convincingly pretend that only processes are to blame. Remember that the current thinking is working against thousands of years of a cultural pattern assigning responsibility. It's also interesting how many seem to be using this idea meant to be applied to internal processes and individuals and applying it to the companies themselves. The most significant example was Gitlab: they had something like four backup mechanisms for their database, yet three(!) of them had not been working for months, with nobody noticing. Then, someone actually dropped the production database. The last working backup system also had some problems I can't remember right now. Yet in the comments here, there was barely a hint of criticism of Gitlab. Instead, they were lauded for not lynching the poor guy who made the last of what must have been dozens of terrible decisions leading to the incident. Travis' incident here does actually seem like the sort of freak accident that one can't rule out completely, at least not at a company of their size. But forgiveness might not always be warranted.

detaro 9y ago
> Yet in the comments here, there was barely a hint of criticism of Gitlab. Instead, they were lauded for not lynching the poor guy who made the last of what must have been dozens of terrible decisions leading to the incident. The threads are still there. You can go back and read them, and notice that quite a few comments are along the lines "not an individual fault, but catastrophic on the group level", "I don't think I can trust Gitlab to run a production service", "How the hell do you have all your backups failing and not notice?!".
bshimmin 9y ago
Isn't the fact that there were "dozens of terrible decisions" leading to one poor guy dropping the production database (in a way that was extremely hard to recover) absolutely a failure of process rather than a failure of a person? He made the last mistake in a big series of mistakes, so unless he was also responsible for all the prior mistakes and failed/non-existent processes leading to his mistake, it seems very unfair to blame him. It is, of course, perfectly fair to blame Gitlab, and plenty of people did just that in the comment thread about it at the time.
solatic 9y ago
> I think when the circumstances actually show someone was grossly negligent, it's very hard to convincingly pretend that only processes are to blame. On the contrary - the entire reason why you transition from culture to process as you grow is because when you measure by results, there's no difference between unintentional negligence (leaving a production terminal open à la Travis), intentional negligence (let me bystep this annoying check, let me procrastinate on fixing the backups a la GitLab), and malice. If you get large enough, you will have grossly negligent people, by statistical inevitability. You can either accept this statistical inevitability and design your process for it or you can continue to believe that you (and everybody else) really actually do only ever hire the very best.
- subway 9y ago
  GP is pretty clearly talking about blaming the company (processes) and not individuals. The entire post laments the fact that entities external to the company will repeatedly forgive the company if the company goes through the motion of publishing a blameless post-mortem. Even if those post-mortems repeatedly indicate the company really didn't learn anything, or change the processes that led up to the incident. But by golly, they're forward thinking company, and therefore blameless.

detaro 9y ago

> Yet in the comments here, there was barely a hint of criticism of Gitlab. Instead, they were lauded for not lynching the poor guy who made the last of what must have been dozens of terrible decisions leading to the incident. The threads are still there. You can go back and read them, and notice that quite a few comments are along the lines "not an individual fault, but catastrophic on the group level", "I don't think I can trust Gitlab to run a production service", "How the hell do you have all your backups failing and not notice?!".

bshimmin 9y ago

Isn't the fact that there were "dozens of terrible decisions" leading to one poor guy dropping the production database (in a way that was extremely hard to recover) absolutely a failure of process rather than a failure of a person? He made the last mistake in a big series of mistakes, so unless he was also responsible for all the prior mistakes and failed/non-existent processes leading to his mistake, it seems very unfair to blame him. It is, of course, perfectly fair to blame Gitlab, and plenty of people did just that in the comment thread about it at the time.

solatic 9y ago

> I think when the circumstances actually show someone was grossly negligent, it's very hard to convincingly pretend that only processes are to blame. On the contrary - the entire reason why you transition from culture to process as you grow is because when you measure by results, there's no difference between unintentional negligence (leaving a production terminal open à la Travis), intentional negligence (let me bystep this annoying check, let me procrastinate on fixing the backups a la GitLab), and malice. If you get large enough, you will have grossly negligent people, by statistical inevitability. You can either accept this statistical inevitability and design your process for it or you can continue to believe that you (and everybody else) really actually do only ever hire the very best.

subway 9y ago
GP is pretty clearly talking about blaming the company (processes) and not individuals. The entire post laments the fact that entities external to the company will repeatedly forgive the company if the company goes through the motion of publishing a blameless post-mortem. Even if those post-mortems repeatedly indicate the company really didn't learn anything, or change the processes that led up to the incident. But by golly, they're forward thinking company, and therefore blameless.