4 ms·

Somebody in my company calculated that 70% of the incidents at my company happened due to pushy, penalizing management forcing engineers to rush things out. In

Somebody in my company calculated that 70% of the incidents at my company happened due to pushy, penalizing management forcing engineers to rush things out. Incident reviews literally pointed out "being rushed".

listenallyall 3y ago
To be fair, do "incident reviews" mean anything? Most people blame someone else when things go south. How many people would complete a survey with, "Honestly, I'm not a very good programmer, I had no idea this piece of code was exploit-able."
- ochronus 3y ago
  Yes, they do – if done well. These are supposed to be blameless postmortems, focusing on what we can do better in the system to prevent similar issues from happening in the future. The answer is never “get better programmers”. Typical action items from the couple of recent postmortems I led/were a part of were along the lines of “decrease this alert's threshold, so we know about it earlier” ; “we need to invest in a proper local testing environment” ; “let's make our service independent of that other service” etc.
- MadnessASAP 3y ago
  A particular bit that I appreciate about my workplaces "incident review" process is that it is anonymous and non-punitave. They exist solely to understand the root causes that lead to an incident. They also don't stop asking why, an employee did something stupid? Why? Were they not properly trained? Why not? Were they pressured? By whom and why? Why did they pressure the first? Etc, etc. They're can of course be a parallel disciplinary investigation of the situation so requires, but they don't share notes and you're not obligated to tell them both the same story. It's a model I wish more organizations would adopt. People make mistakes or deliberately do things that are wrong for all kinds of reasons. Pretending they don't or won't is a poor plan.
- piva00 3y ago
  I have written my fair share of post-mortems with a conclusion of "my knowledge of the topic/domain was limited at the time of implementation and hence created issue X due to Y". It all depends on the organisation's culture, if management is always looking for blame you'll get meaningless incident reviews caused by fear, if there's trust you'll get a larger share of honest reviews with actionable points to minimising the risk of it happening again.
koliber 3y ago
That is probably true. Such incidents are one of the costs of rushing things out to market. What is the share of newly created profits and seized market opportunities due to rushing things out? If the latter is greater then the former, in some quantifiable sense, the company might be doing alright.
- nine_zeros 3y ago
  > What is the share of newly created profits and seized market opportunities due to rushing things out? If the latter is greater then the former, in some quantifiable sense, the company might be doing alright. The problem is not what is the cost of the incidents themselves to the business. The problem is how management deals with it. In my company, they usually spend an enormous amount of time in incident reviews, trying to find the blame (usually engineers), adding more processes, firing someone, spending 6 months and $$ backfilling the role. The true cost of poor management is often hidden in bureaucracy.
moribvndvs 3y ago
Literally just happened. We advised that a change had systemic impact and we needed a proper regression. Mgmt said the build addresses an urgent issue (there was a bug that caught the attention of a CEO from partner and they were squeezing the vice on our balls) and it’s their call and responsibility, but get it out there. Unfortunately, the release did cause problems, causing much (rightful) anger from our customers. We rolled it back. Now engineering is subject to a witch hunt from senior management about quality concerns. Management who authorized is currently wearing a hot dog costume demanding to know who crashed the hot dog themed car through the wall. It’s always the same shit.
- nine_zeros 3y ago
  > Now engineering is subject to a witch hunt from senior management about quality concerns. Management who authorized is currently wearing a hot dog costume demanding to know who crashed the hot dog themed car through the wall. Management: "Roll-out fast, fast, fast so that I can look good. But if something fails in this fast roll out, we will find a way to beat it out of engineers."

listenallyall 3y ago

To be fair, do "incident reviews" mean anything? Most people blame someone else when things go south. How many people would complete a survey with, "Honestly, I'm not a very good programmer, I had no idea this piece of code was exploit-able."

ochronus 3y ago
Yes, they do – if done well. These are supposed to be blameless postmortems, focusing on what we can do better in the system to prevent similar issues from happening in the future. The answer is never “get better programmers”. Typical action items from the couple of recent postmortems I led/were a part of were along the lines of “decrease this alert's threshold, so we know about it earlier” ; “we need to invest in a proper local testing environment” ; “let's make our service independent of that other service” etc.
MadnessASAP 3y ago
A particular bit that I appreciate about my workplaces "incident review" process is that it is anonymous and non-punitave. They exist solely to understand the root causes that lead to an incident. They also don't stop asking why, an employee did something stupid? Why? Were they not properly trained? Why not? Were they pressured? By whom and why? Why did they pressure the first? Etc, etc. They're can of course be a parallel disciplinary investigation of the situation so requires, but they don't share notes and you're not obligated to tell them both the same story. It's a model I wish more organizations would adopt. People make mistakes or deliberately do things that are wrong for all kinds of reasons. Pretending they don't or won't is a poor plan.
piva00 3y ago
I have written my fair share of post-mortems with a conclusion of "my knowledge of the topic/domain was limited at the time of implementation and hence created issue X due to Y". It all depends on the organisation's culture, if management is always looking for blame you'll get meaningless incident reviews caused by fear, if there's trust you'll get a larger share of honest reviews with actionable points to minimising the risk of it happening again.

koliber 3y ago

That is probably true. Such incidents are one of the costs of rushing things out to market. What is the share of newly created profits and seized market opportunities due to rushing things out? If the latter is greater then the former, in some quantifiable sense, the company might be doing alright.

nine_zeros 3y ago
> What is the share of newly created profits and seized market opportunities due to rushing things out? If the latter is greater then the former, in some quantifiable sense, the company might be doing alright. The problem is not what is the cost of the incidents themselves to the business. The problem is how management deals with it. In my company, they usually spend an enormous amount of time in incident reviews, trying to find the blame (usually engineers), adding more processes, firing someone, spending 6 months and $$ backfilling the role. The true cost of poor management is often hidden in bureaucracy.

moribvndvs 3y ago

Literally just happened. We advised that a change had systemic impact and we needed a proper regression. Mgmt said the build addresses an urgent issue (there was a bug that caught the attention of a CEO from partner and they were squeezing the vice on our balls) and it’s their call and responsibility, but get it out there. Unfortunately, the release did cause problems, causing much (rightful) anger from our customers. We rolled it back. Now engineering is subject to a witch hunt from senior management about quality concerns. Management who authorized is currently wearing a hot dog costume demanding to know who crashed the hot dog themed car through the wall. It’s always the same shit.

nine_zeros 3y ago
> Now engineering is subject to a witch hunt from senior management about quality concerns. Management who authorized is currently wearing a hot dog costume demanding to know who crashed the hot dog themed car through the wall. Management: "Roll-out fast, fast, fast so that I can look good. But if something fails in this fast roll out, we will find a way to beat it out of engineers."