4 ms·
I agree that the ultimate root fault isn’t the recent data entry. > At a private company a failure like this would see the CTO immediately fired and even the C
by sokoloff 4y ago
I agree that the ultimate root fault isn’t the recent data entry.
> At a private company a failure like this would see the CTO immediately fired and even the CEO having to beg the board for forgiveness and keeping their job.
That part doesn’t match my experience at all.
- toomuchtodo 4y agoRight. Engineer closest to the fault gets scapegoated and fired, new hire gets hired in without any idea what they’re coming into, and the show goes on. Good interview question to ask: “am I replacing a recent departure? If so, why?” You either get the truth or deception signal once hired. Both are helpful to know, and asking the question is free.
- marcosdumay 4y ago> Engineer closest to the fault gets scapegoated and fired That happens. But not every time, and even "usually" requires some statistics collecting to be sustained. Very often people just say "yes, we learned an expensive lesson", and change something that may or may not help the next time it happens.
- toomuchtodo 4y agoReally just depends on the org, and you’re not going to know until you’re on the inside.
- marcosdumay 4y agoYep, I completely agree with that. It can change from team to team too. Also, it can suddenly change at any time without a warning.
- hermitdev 4y agoI think it also matters on what the response to the incident was, in particular the answer to the questions of: who was responsible and how can we prevent this from happening again? If the answer to the first question is a bunch of finger pointing or the answer to the second is along the lines of "nothing", heads will likely roll. My first boss as a FTE at a trading shop made a change to a production database outside of process before a weekend. Come Monday morning, production trading systems couldn't come up for market open. We were also an options market maker. He refused to accept responsibility for his actions and denied there was anything done to prevent it. He was quickly behind closed doors in his boss's office. I had to undo the changes he made the week before (hooray for audit tables) as soon as I could. We were potentially liable for a substantial penalty from the exchange for being out of the market, and every minute mattered. Don't know if we ever did get penalized, but he was fired and out the door by 10AM that day. I've certainly seen and made lots of mistakes over the decades since, but that's the only one for which I've seen someone canned. Response matters.
- drstewart 4y ago>Right. Engineer closest to the fault gets scapegoated and fired, new hire gets hired in without any idea what they’re coming into, and the show goes on. That part doesn’t match my experience at all.
- pleb_nz 4y agoYou can just about tell what parts of the world a poster comes from when they start throwing the word 'fire' around willy nilly. Good question though..
- KennyBlanken 4y agoYou can tell what part of the world a poster comes from when they're smugly pretending that companies still don't have plenty of levers to pull to push you out the door, doing so by making you miserable. Oh, and the rampant racism and classism, thanks to the standard practice of including a photo on one's resume...
- scruple 4y agoSame here. In the places I've experienced with poor management and engineering cultures, what I've seen is management dictates all sorts of horrible, asinine things (like story points being treated as deadlines, etc.) that leads directly to problems like this because engineering is not given the space it needs to create robust solutions in the first place. The prevailing winds are, "Just get it done." The outcome is constant fire fighting and triage because of simple problems like bad data finding it's way into the system.
- drbeast 4y ago[dead]
- bennyelv 4y agoIt might not be the CTO's fault at all - maybe they inherited it, identified the issues, and are getting them solved. Probably not in this case, but you couldn't assume for every...
- chiefalchemist 4y agoFunny. I'm reading The Phoenix Project and just finished the about the CEO chewing out various employees. I keep thinking, responsiblity flows upstream, or should. Ultimately it is the CEO's job to put his people and the project in a position to suceed. Maybe the CEO in the book gets his due? Blaming a contractor? To save face? Isn't saving face. It's an embarrassment.
- noughtme 4y agoStill looks interesting, but I was hoping The Phoenix Project was a post-mortem of the Phoenix Pay System (scandal) https://en.m.wikipedia.org/wiki/Phoenix_pay_system https://en.m.wikipedia.org/wiki/Phoenix_pay_system
- chiefalchemist 4y agoIt may be "inspired by" as in the book TPP is among other things about the stores POS. TBH IDK the book was mentioned on HN and I figured I'd check it out.
- fineIllregister 4y ago> That part doesn’t match my experience at all. Exactly! There wasn't any real fallout for Southwest, who had a shutdown less than a month ago, for similar outdated technology reasons.
- Someone1234 4y agoMaybe even more pertinent is that they had a similar one in 2016 (smaller scale, but similar cause/effect) and then didn't fix anything, so it could happen again recently. There's running lean and then there is running recklessly.
- CamperBob2 4y ago"Move slowly and break things!"