6 ms·

I was responsible for a software bug that forced the entire United States Coast Guard to revert to using paper records to track all of their Aviation Logistics

I was responsible for a software bug that forced the entire United States Coast Guard to revert to using paper records to track all of their Aviation Logistics for an entire month. The bug was in an aircraft maintenance dashboard that had over a dozen filters on it. It passed a thorough round of QAing, but once in production it was found that a specific selection of filters would create an infinite loop in the report that would crash the system. With hundreds of mechanics using the dashboard, someone was triggering that selection of filters every two to three days. It was a very rough month for me. When an emergency patch failed to solve the issue, management decided to take the whole system down (it was too big a release and had been in production too long to revert). User satisfaction surveys bottomed-out. There was discussion in upper-management about firing me, but my immediate supervisor defended me and pointed out that this required failure at many levels. When we got the system back online, they had to hire dozens of contractors to manually enter all the paper records into the system. I continued to code for the Coast Guard for five more years, got a big promotion to Senior Developer at one point, and left the organization with much respect. Every once in a awhile someone would present a graph of user satisfaction surveys over the years and there would be that awful month where the graph bottomed out, but I could eventually smile about what that junior developer did.

kadendogthing 8y ago
Huge kudos to your manager for standing up like that. Companies often look for a single point of failure (it's only human), instead of admitting that the organization as a whole systematically failed to prevent the error, let alone discover it "in time." I hate to use platitudes but companies are only really as strong as their weakest link (not to impugn your skills or anything like that). Letting a single failure sink the whole ship means there are larger issues at play.
- _____bee_____ 8y ago
  `Work for people who can put trust in you, and can fight for you. ` a lesson I learn the hard way as well
- tabtab 8y ago
  Re: Huge kudos to your manager for standing up like that. Companies often look for a single point of failure Indeed. Orgs like simplistic stories and simplistic blame (scapegoats). They'd like to fire somebody(s) and say, "we solved it all." Anything important/critical should have multiple eyeballs vetting it, and mistakes do happen. If they don't have multiple eyeballs vetting it, then the people who failed to hire enough should be given a stern warning. I don't believe in firing for one mistake, unless it's egregious. People get better as they learn from mistakes.
- crimsonalucard 8y ago
  You also need to be a person worth fighting for. It goes both ways.
  - kadendogthing 8y ago
    Presumably that's what the hiring process is for.
    - crimsonalucard 8y ago
      Right. Everyone knows that the hiring process is perfect and accurate and that if you're hired there's no need to prove yourself because you already did that in the interview are you kidding me?
      - kadendogthing 8y ago
        I think you have a very maligned view of what managers should be if you agree with the thought that people need to prove themselves to managers to be "fought for." Especially over a mistake that multiple layers of the organization didn't catch and weren't equipped to handle. No I'm not kidding you. That view point seems either very naive or nonconstructive. And considering you had to resort to a disingenuous straw man in order to respond to something I didn't say or imply, I'd say the latter is more likely.
        crimsonalucard 8y ago
        You need to constantly prove yourself everyday. Lets put it this way. If I don't show up for work and I did well on the interview. I'm not proving myself aren't I? If I do horrible work, if my code is buggy and full of errors then what am I? No manager will fight for that despite a strong interview. This is simple logic and common sense. Obviously the person in the original posting proved himself.
notafrog 8y ago
Wow...I can't even begin to imagine how awful you must have felt. But, in hindsight, it's really cool too, in a weird way.

kadendogthing 8y ago

Huge kudos to your manager for standing up like that. Companies often look for a single point of failure (it's only human), instead of admitting that the organization as a whole systematically failed to prevent the error, let alone discover it "in time." I hate to use platitudes but companies are only really as strong as their weakest link (not to impugn your skills or anything like that). Letting a single failure sink the whole ship means there are larger issues at play.

_____bee_____ 8y ago
`Work for people who can put trust in you, and can fight for you. ` a lesson I learn the hard way as well
tabtab 8y ago
Re: Huge kudos to your manager for standing up like that. Companies often look for a single point of failure Indeed. Orgs like simplistic stories and simplistic blame (scapegoats). They'd like to fire somebody(s) and say, "we solved it all." Anything important/critical should have multiple eyeballs vetting it, and mistakes do happen. If they don't have multiple eyeballs vetting it, then the people who failed to hire enough should be given a stern warning. I don't believe in firing for one mistake, unless it's egregious. People get better as they learn from mistakes.
crimsonalucard 8y ago
You also need to be a person worth fighting for. It goes both ways.
- kadendogthing 8y ago
  Presumably that's what the hiring process is for.
  - crimsonalucard 8y ago
    Right. Everyone knows that the hiring process is perfect and accurate and that if you're hired there's no need to prove yourself because you already did that in the interview are you kidding me?
    - kadendogthing 8y ago
      I think you have a very maligned view of what managers should be if you agree with the thought that people need to prove themselves to managers to be "fought for." Especially over a mistake that multiple layers of the organization didn't catch and weren't equipped to handle. No I'm not kidding you. That view point seems either very naive or nonconstructive. And considering you had to resort to a disingenuous straw man in order to respond to something I didn't say or imply, I'd say the latter is more likely.
      - crimsonalucard 8y ago
        You need to constantly prove yourself everyday. Lets put it this way. If I don't show up for work and I did well on the interview. I'm not proving myself aren't I? If I do horrible work, if my code is buggy and full of errors then what am I? No manager will fight for that despite a strong interview. This is simple logic and common sense. Obviously the person in the original posting proved himself.

notafrog 8y ago

Wow...I can't even begin to imagine how awful you must have felt. But, in hindsight, it's really cool too, in a weird way.