3 ms·

This is just a failure in change management. They detected the issue in 4 minutes but it took over an hour and a half to mitigate?

gonzo41 9y ago
1 minutes to fix. 30 minutes rationalizing that the 1 minute fix your thinking of is going to work. 45 minutes convincing others. 14 minutes of terrible self doubt and worry. And we're back up. Ops life!
- greenleafjacob 9y ago
  Mitigate first. If you are confident in your change management system then rolling back is always safe.
SlowBro 9y ago
And quite possibly a failure in disaster recovery methods. If it was at all possible to have a replicated site it should have been failed over to it immediately. Then resolution and root cause analysis could have been carried over at their leisure. Perhaps that would not have been possible with this style of equipment, I don’t know. But at least yes, change management procedures should have been in place and obeyed. However, I hope they don’t fire the guy. I once read of an employee of some company who created a $600k mistake. They asked the CEO if he would fire the employee. “I just paid $600k to educate one of my employees. Do you think I’m going to give him away to my competitors?” That has stuck with me. Unless there is clearly malicious intent I want to give second chances.
bonesss 9y ago
That's the conclusion of the report: no one was previously aware that an "unrelated" field in an "unrelated" activity would have production-wide consequences. The detected an issue within 4 minutes, but it took an hour and a half to diagnose the issue, find the unwanted and unforeseen change, and revert it. That's not an absurd amount of time for a complex system with millions of users. I would hazard to guess that this long-running system isn't even taken in as part of the companies 'change management' routines, in so far as it's job was number filtering and the operation was 'routine'. At my work we produce production change reports for production changes, but we don't fill them out every time we run a stable application...
jlgaddis 9y ago
Four minutes to realize SHTF, but a while longer to figure out why: > Level 3 was aware it had a problem within four minutes, the FCC report said. The problem was difficult to diagnose, however, because no one at Level 3 was aware of the consequences of leaving that particular field empty, nor had anyone at the company previously seen the system behave the way it was behaving. That is, they didn't know that leaving that field blank is what caused the S to HTF.
- Dodgeit 9y ago
  I can imagine it once they found out the cause. "Really? That was it? Are you fucking serious?"
  - avs733 9y ago
    sounds like most of the manufacturing issues I have been involved in... 'why did the machine break?' 'well the spec says to clean it using chemical X but the cabinet with X is 75 feet away and the cabinet with chemical Y is next to the machine so they used Y. They use X and Y interchangeably on other machine so the technicians (note: high-school grads, great guys but not chemists) thought it was interchangeable on this machine.' 'well why aren't they interchangeable on this machine' 'Y reacts with the glue used to assemble the machine, which was a change in the newer versions because of EPA regulations, so doing this weekly maintenance task for 8 years was finally enough to degrade the glue' 'so why was chemical Y stored near here?' 'because X has to be kept so many feet from Y. Last year in the efficiency audit we found techs had to walk to far on average to get X so we moved the X cabinet, which resulted in us moving Y to here.' 'has Y been used on any of the other machines when it shouldn't have been?' 'we don't know and we aren't sure how to check' Fault trees usually make for really interesting reading.
    - exikyut 9y ago
      Nice. I'm guessing I can't read the source to this particular fault tree, but I wonder where I might find others. Preferably without digging through e.g. troves of court documents and the like.
hn20180220 9y ago
At most organizations change management is nothing but bureaucracy that does nothing but preventing things from getting done.

gonzo41 9y ago

1 minutes to fix. 30 minutes rationalizing that the 1 minute fix your thinking of is going to work. 45 minutes convincing others. 14 minutes of terrible self doubt and worry. And we're back up. Ops life!

greenleafjacob 9y ago
Mitigate first. If you are confident in your change management system then rolling back is always safe.

SlowBro 9y ago

And quite possibly a failure in disaster recovery methods. If it was at all possible to have a replicated site it should have been failed over to it immediately. Then resolution and root cause analysis could have been carried over at their leisure. Perhaps that would not have been possible with this style of equipment, I don’t know. But at least yes, change management procedures should have been in place and obeyed. However, I hope they don’t fire the guy. I once read of an employee of some company who created a $600k mistake. They asked the CEO if he would fire the employee. “I just paid $600k to educate one of my employees. Do you think I’m going to give him away to my competitors?” That has stuck with me. Unless there is clearly malicious intent I want to give second chances.

bonesss 9y ago

That's the conclusion of the report: no one was previously aware that an "unrelated" field in an "unrelated" activity would have production-wide consequences. The detected an issue within 4 minutes, but it took an hour and a half to diagnose the issue, find the unwanted and unforeseen change, and revert it. That's not an absurd amount of time for a complex system with millions of users. I would hazard to guess that this long-running system isn't even taken in as part of the companies 'change management' routines, in so far as it's job was number filtering and the operation was 'routine'. At my work we produce production change reports for production changes, but we don't fill them out every time we run a stable application...

jlgaddis 9y ago

Four minutes to realize SHTF, but a while longer to figure out why: > Level 3 was aware it had a problem within four minutes, the FCC report said. The problem was difficult to diagnose, however, because no one at Level 3 was aware of the consequences of leaving that particular field empty, nor had anyone at the company previously seen the system behave the way it was behaving. That is, they didn't know that leaving that field blank is what caused the S to HTF.

Dodgeit 9y ago
I can imagine it once they found out the cause. "Really? That was it? Are you fucking serious?"
- avs733 9y ago
  sounds like most of the manufacturing issues I have been involved in... 'why did the machine break?' 'well the spec says to clean it using chemical X but the cabinet with X is 75 feet away and the cabinet with chemical Y is next to the machine so they used Y. They use X and Y interchangeably on other machine so the technicians (note: high-school grads, great guys but not chemists) thought it was interchangeable on this machine.' 'well why aren't they interchangeable on this machine' 'Y reacts with the glue used to assemble the machine, which was a change in the newer versions because of EPA regulations, so doing this weekly maintenance task for 8 years was finally enough to degrade the glue' 'so why was chemical Y stored near here?' 'because X has to be kept so many feet from Y. Last year in the efficiency audit we found techs had to walk to far on average to get X so we moved the X cabinet, which resulted in us moving Y to here.' 'has Y been used on any of the other machines when it shouldn't have been?' 'we don't know and we aren't sure how to check' Fault trees usually make for really interesting reading.
  - exikyut 9y ago
    Nice. I'm guessing I can't read the source to this particular fault tree, but I wonder where I might find others. Preferably without digging through e.g. troves of court documents and the like.

hn20180220 9y ago

At most organizations change management is nothing but bureaucracy that does nothing but preventing things from getting done.