4 ms·

I would argue that someone probably deserves to be fired for this. It's obviously not the guy that pushed the button nor the guy that wrote the software. Howeve

I would argue that someone probably deserves to be fired for this. It's obviously not the guy that pushed the button nor the guy that wrote the software. However, there was, probably, a guy that several years ago didn't do his job well - his job was to make sure that this kind of errors cannot happen, unless there is an intentional malicious action of course. We see it all the time in IT - people totally misjudge the 'risk'. 1% risk of breakdown quite often seems acceptable to decision makers until it actually happens. Then all hell breaks loose.

Matt3o12_ 9y ago
There is hardly just one person responsible for doing this. There was at least one developer who implemented this solution, at least “designer” that can up with a solution for the initial problem, a dozen managers who oversaw that whole project, up to the minister who must have been aware of this problem. Let alone everybody who ever worked with this system and never reported that “bug”. So if you want to fire someone, start at the top for not overseeing this correctly or hiring the wrong person for the job and then work your way to the bottom. I believe it is much better to fix it (require a second authentication before sending those massages) and search for similar problems within the whole department to make sure something like this never happens again. Make sure everybody, including new hires, are aware of this incident, what the consequences were and that everybody is encouraged to report similar bugs and nobody has to be afraid to be fired when they make a mistake. Otherwise you just create an environment where everybody blames anybody but themselves and the “smallest” guy gets fired for mistakes other’s. This way, things get never fixed probably because everyone ist just afraid of the consequences.
- rickycook 9y ago
  i’m not so sure there would have been any kind of designer. from my dealings with govt projects, especially internal ones, they rarely do any UX of any kind. the interface is ad-hoc from the devs, and you can’t expect a dev to design a fool proof interface. if you direct the blame 1 level further, then it was whoever was responsive for budget/hiring/spec that didn’t ensure that there was a UX designer on the project, but at the same time you can kinda understand why not: the value of UX is frequently not understood, especially when people think “it’s just internal; why make it easy to use if only we see it?”. the answer of course is that you pay for internal systems every day in employee time. anyway, the point i guess is you can probably trace the “blame” back far enough that it’s too far removed to say it’s anyone’s fault per se, but it proves the point that you need design across the board, and that’s frequently not well understood
spc476 9y ago
A few years ago we had an outage that cost us quite a bit of money [1]. The root cause was the deletion of a test record from the database we query (and run by the Oligarchist Phone Company) but the deletion had a domino effect on our service because of the way the error was treated by our side (the test record was never supposed to be deleted because it's used to ensure the database we query is up and running [2][3]). From what I understand, we were told the record would never get removed, and we relied upon that. Could our error handling been better? Yes. Could the customer's procedures been better to prevent the record from being deleted? Yes. It was a mistake on both parties. Who should have gotten fired? [1] Very scary SLAs with the Oligarchist Phone Company. [2] It's actually a DNS query. NAPTR records to be precise. Search on that if you are interested. [3] It was also important because that was the only record we could query and not get charged for it. Everything in the telephony world cross-company (and even cross-department) gets charged. Somebody makes money; somebody pays money.
bonestamp2 9y ago
What does firing that person achieve?

Matt3o12_ 9y ago

There is hardly just one person responsible for doing this. There was at least one developer who implemented this solution, at least “designer” that can up with a solution for the initial problem, a dozen managers who oversaw that whole project, up to the minister who must have been aware of this problem. Let alone everybody who ever worked with this system and never reported that “bug”. So if you want to fire someone, start at the top for not overseeing this correctly or hiring the wrong person for the job and then work your way to the bottom. I believe it is much better to fix it (require a second authentication before sending those massages) and search for similar problems within the whole department to make sure something like this never happens again. Make sure everybody, including new hires, are aware of this incident, what the consequences were and that everybody is encouraged to report similar bugs and nobody has to be afraid to be fired when they make a mistake. Otherwise you just create an environment where everybody blames anybody but themselves and the “smallest” guy gets fired for mistakes other’s. This way, things get never fixed probably because everyone ist just afraid of the consequences.

rickycook 9y ago
i’m not so sure there would have been any kind of designer. from my dealings with govt projects, especially internal ones, they rarely do any UX of any kind. the interface is ad-hoc from the devs, and you can’t expect a dev to design a fool proof interface. if you direct the blame 1 level further, then it was whoever was responsive for budget/hiring/spec that didn’t ensure that there was a UX designer on the project, but at the same time you can kinda understand why not: the value of UX is frequently not understood, especially when people think “it’s just internal; why make it easy to use if only we see it?”. the answer of course is that you pay for internal systems every day in employee time. anyway, the point i guess is you can probably trace the “blame” back far enough that it’s too far removed to say it’s anyone’s fault per se, but it proves the point that you need design across the board, and that’s frequently not well understood

spc476 9y ago

A few years ago we had an outage that cost us quite a bit of money [1]. The root cause was the deletion of a test record from the database we query (and run by the Oligarchist Phone Company) but the deletion had a domino effect on our service because of the way the error was treated by our side (the test record was never supposed to be deleted because it's used to ensure the database we query is up and running [2][3]). From what I understand, we were told the record would never get removed, and we relied upon that. Could our error handling been better? Yes. Could the customer's procedures been better to prevent the record from being deleted? Yes. It was a mistake on both parties. Who should have gotten fired? [1] Very scary SLAs with the Oligarchist Phone Company. [2] It's actually a DNS query. NAPTR records to be precise. Search on that if you are interested. [3] It was also important because that was the only record we could query and not get charged for it. Everything in the telephony world cross-company (and even cross-department) gets charged. Somebody makes money; somebody pays money.

bonestamp2 9y ago

What does firing that person achieve?