4 ms·

Indeed! What a great way to see failure: what went wrong, how to improve without blaming it on the poor guy. I wish all companies were like this - very forward

by mk89 9y ago

Indeed! What a great way to see failure: what went wrong, how to improve without blaming it on the poor guy. I wish all companies were like this - very forward thinking! Funny thing is... this morning on my commute I was watching https://www.oreilly.com/ideas/developing-a-chaos-architecture-mindset https://www.oreilly.com/ideas/developing-a-chaos-architectur... - which I recommend to anyone!

twic 9y ago
I think it is now widely understood that best practice is to blame processes, not people. However, i do have a niggling worry that letting people off the hook risks stunting their personal growth. If you can make mistakes and not bear any responsibility for it, what is driving you to get better at doing your job?
- oblio 9y ago
  In any well run engineering org there is still the grapevine in operation. You really don't want to be the butt of all the "Joe the prod database killer" jokes.
- IAmEveryone 9y ago
  I think when the circumstances actually show someone was grossly negligent, it's very hard to convincingly pretend that only processes are to blame. Remember that the current thinking is working against thousands of years of a cultural pattern assigning responsibility. It's also interesting how many seem to be using this idea meant to be applied to internal processes and individuals and applying it to the companies themselves. The most significant example was Gitlab: they had something like four backup mechanisms for their database, yet three(!) of them had not been working for months, with nobody noticing. Then, someone actually dropped the production database. The last working backup system also had some problems I can't remember right now. Yet in the comments here, there was barely a hint of criticism of Gitlab. Instead, they were lauded for not lynching the poor guy who made the last of what must have been dozens of terrible decisions leading to the incident. Travis' incident here does actually seem like the sort of freak accident that one can't rule out completely, at least not at a company of their size. But forgiveness might not always be warranted.
  - detaro 9y ago
    > Yet in the comments here, there was barely a hint of criticism of Gitlab. Instead, they were lauded for not lynching the poor guy who made the last of what must have been dozens of terrible decisions leading to the incident. The threads are still there. You can go back and read them, and notice that quite a few comments are along the lines "not an individual fault, but catastrophic on the group level", "I don't think I can trust Gitlab to run a production service", "How the hell do you have all your backups failing and not notice?!".
  - bshimmin 9y ago
    Isn't the fact that there were "dozens of terrible decisions" leading to one poor guy dropping the production database (in a way that was extremely hard to recover) absolutely a failure of process rather than a failure of a person? He made the last mistake in a big series of mistakes, so unless he was also responsible for all the prior mistakes and failed/non-existent processes leading to his mistake, it seems very unfair to blame him. It is, of course, perfectly fair to blame Gitlab, and plenty of people did just that in the comment thread about it at the time.
  - solatic 9y ago
    > I think when the circumstances actually show someone was grossly negligent, it's very hard to convincingly pretend that only processes are to blame. On the contrary - the entire reason why you transition from culture to process as you grow is because when you measure by results, there's no difference between unintentional negligence (leaving a production terminal open à la Travis), intentional negligence (let me bystep this annoying check, let me procrastinate on fixing the backups a la GitLab), and malice. If you get large enough, you will have grossly negligent people, by statistical inevitability. You can either accept this statistical inevitability and design your process for it or you can continue to believe that you (and everybody else) really actually do only ever hire the very best.
    - subway 9y ago
      GP is pretty clearly talking about blaming the company (processes) and not individuals. The entire post laments the fact that entities external to the company will repeatedly forgive the company if the company goes through the motion of publishing a blameless post-mortem. Even if those post-mortems repeatedly indicate the company really didn't learn anything, or change the processes that led up to the incident. But by golly, they're forward thinking company, and therefore blameless.
- mk89 9y ago
  > If you can make mistakes and not bear any responsibility for it, what is driving you to get better at doing your job? It could be any of the following: 1) ambition: desire to improve yourself 2) fear: desire not to screw up (and feel that you are making mistakes) 3) fear: job security In all cases what's essential is a feedback system. The fact that you are not directly the one to be blamed doesn't imply you are automatically not accountable for anything you do. Everyone shares at least partially the blame. Creating a blameless culture, having a good feedback system in place with maybe some goals, objectives and ways to improve is probably something you need, if you want to keep your employees and have them grow with you and not treat them as expandable. Good management should be able to do this. If someone is not fit for a role, a good manager will be able to find another way for this person to be useful. The last thing you want to do is to fire someone. At least that's my opinion. Again, management.
- rtpg 9y ago
  While in a huge company it might be hard, I feel like any small engineering team can be composed entirely of self motivated people If you make a mistake (a real one that causes problems for the company and its users), that is enough for most people to want to avoid it happening again.
- haggy 9y ago
  IMO it's distinguishing mistakes from pure negligence. A developer who's able to easily run scripts in production without any checks in place is different than a developer who knowingly sidesteps existing processes in order to perform a task faster or easier. The punishment for one versus the other should be very different.
- osteele 9y ago
  These things can be different: whether you take personal responsibility for a mistake, whether you are legally responsible for a mistake, whether your manager holds you responsible for a mistake, whether your team holds you responsible for a mistake, and whether a post-mortem report blames you for a mistake. It may be easier to take personal responsibility for a mistake if this isn't coupled with these other things.

by mk89 9y ago

twic 9y ago

I think it is now widely understood that best practice is to blame processes, not people. However, i do have a niggling worry that letting people off the hook risks stunting their personal growth. If you can make mistakes and not bear any responsibility for it, what is driving you to get better at doing your job?

oblio 9y ago
In any well run engineering org there is still the grapevine in operation. You really don't want to be the butt of all the "Joe the prod database killer" jokes.
IAmEveryone 9y ago
I think when the circumstances actually show someone was grossly negligent, it's very hard to convincingly pretend that only processes are to blame. Remember that the current thinking is working against thousands of years of a cultural pattern assigning responsibility. It's also interesting how many seem to be using this idea meant to be applied to internal processes and individuals and applying it to the companies themselves. The most significant example was Gitlab: they had something like four backup mechanisms for their database, yet three(!) of them had not been working for months, with nobody noticing. Then, someone actually dropped the production database. The last working backup system also had some problems I can't remember right now. Yet in the comments here, there was barely a hint of criticism of Gitlab. Instead, they were lauded for not lynching the poor guy who made the last of what must have been dozens of terrible decisions leading to the incident. Travis' incident here does actually seem like the sort of freak accident that one can't rule out completely, at least not at a company of their size. But forgiveness might not always be warranted.
- detaro 9y ago
  > Yet in the comments here, there was barely a hint of criticism of Gitlab. Instead, they were lauded for not lynching the poor guy who made the last of what must have been dozens of terrible decisions leading to the incident. The threads are still there. You can go back and read them, and notice that quite a few comments are along the lines "not an individual fault, but catastrophic on the group level", "I don't think I can trust Gitlab to run a production service", "How the hell do you have all your backups failing and not notice?!".
- bshimmin 9y ago
  Isn't the fact that there were "dozens of terrible decisions" leading to one poor guy dropping the production database (in a way that was extremely hard to recover) absolutely a failure of process rather than a failure of a person? He made the last mistake in a big series of mistakes, so unless he was also responsible for all the prior mistakes and failed/non-existent processes leading to his mistake, it seems very unfair to blame him. It is, of course, perfectly fair to blame Gitlab, and plenty of people did just that in the comment thread about it at the time.
- solatic 9y ago
  > I think when the circumstances actually show someone was grossly negligent, it's very hard to convincingly pretend that only processes are to blame. On the contrary - the entire reason why you transition from culture to process as you grow is because when you measure by results, there's no difference between unintentional negligence (leaving a production terminal open à la Travis), intentional negligence (let me bystep this annoying check, let me procrastinate on fixing the backups a la GitLab), and malice. If you get large enough, you will have grossly negligent people, by statistical inevitability. You can either accept this statistical inevitability and design your process for it or you can continue to believe that you (and everybody else) really actually do only ever hire the very best.
  - subway 9y ago
    GP is pretty clearly talking about blaming the company (processes) and not individuals. The entire post laments the fact that entities external to the company will repeatedly forgive the company if the company goes through the motion of publishing a blameless post-mortem. Even if those post-mortems repeatedly indicate the company really didn't learn anything, or change the processes that led up to the incident. But by golly, they're forward thinking company, and therefore blameless.
mk89 9y ago
> If you can make mistakes and not bear any responsibility for it, what is driving you to get better at doing your job? It could be any of the following: 1) ambition: desire to improve yourself 2) fear: desire not to screw up (and feel that you are making mistakes) 3) fear: job security In all cases what's essential is a feedback system. The fact that you are not directly the one to be blamed doesn't imply you are automatically not accountable for anything you do. Everyone shares at least partially the blame. Creating a blameless culture, having a good feedback system in place with maybe some goals, objectives and ways to improve is probably something you need, if you want to keep your employees and have them grow with you and not treat them as expandable. Good management should be able to do this. If someone is not fit for a role, a good manager will be able to find another way for this person to be useful. The last thing you want to do is to fire someone. At least that's my opinion. Again, management.
rtpg 9y ago
While in a huge company it might be hard, I feel like any small engineering team can be composed entirely of self motivated people If you make a mistake (a real one that causes problems for the company and its users), that is enough for most people to want to avoid it happening again.
haggy 9y ago
IMO it's distinguishing mistakes from pure negligence. A developer who's able to easily run scripts in production without any checks in place is different than a developer who knowingly sidesteps existing processes in order to perform a task faster or easier. The punishment for one versus the other should be very different.
osteele 9y ago
These things can be different: whether you take personal responsibility for a mistake, whether you are legally responsible for a mistake, whether your manager holds you responsible for a mistake, whether your team holds you responsible for a mistake, and whether a post-mortem report blames you for a mistake. It may be easier to take personal responsibility for a mistake if this isn't coupled with these other things.