3 ms·
Very early in my career I discovered some mission-critical network devices at the hedge fund I worked at that would have caused a network loop if either of the
by busterarm 3y ago
Very early in my career I discovered some mission-critical network devices at the hedge fund I worked at that would have caused a network loop if either of the two machines were ever rebooted. The two servers were related to some feed data and I had noticed that they were cabled together oddly based on how their function was described to me.
The estimated cost of downtime for these systems was something like $7 million per minute. I had raised the issue to a couple of the staff responsible for the machines and to the networking team but was completely dismissed because "there is no way we would have hooked them up that way" and because I was the FNG.
I then raised the issue again at the weekly group meeting because it seemed important -- somebody was dispatched to check visually and came back to confirm what I said. It was a big deal -- the networking team had about 2 weeks of emergency work to do to resolve the issue cleanly.
EVERYONE was angry at me. Even though I had just averted a catastrophe for the company, I made everyone look bad by doing it and particularly because of my status/position on the team. It was an important lesson learned.
- tehlike 3y agoKeep doing the right thing.
- busterarm 3y agoIt's 20 years later and I've made my career off of pointing out fundamental mistakes made by very smart people. ;) (just with more skill and tact) 90% of what I do is ask dumb questions as if I'm completely clueless.
- tehlike 3y agoI know the feeling. I'm similar in that regard. Debugging complex things by asking dumb questions is extremely fruitful. Also asking dumb questions to newer engs teaches them to think different aspects of a problem themselves. One of the things I'm hoping I could keep alive in my child
- phkahler 3y ago>> EVERYONE was angry at me. Even though I had just averted a catastrophe for the company, I made everyone look bad by doing it and particularly because of my status/position on the team. People can be excessive in both taking credit and placing blame. An appropriate and helpful way to frame this is "the system was configured incorrectly but nobody noticed because the problem never actually happened. It's a good thing the new guy had time to go through things and spotted the problem before it ever happened." No need to crucify the team or exaggerate the value of the new guy.
- mablopoule 3y agoOn his book "The Secrets of Consulting", Gerald Weinberg advise against improving more than 10% of performance, and if so, of hoping to have any credit. Just like the article, his reasoning is that if you improve performance too much, it makes management/the team look bad for not doing it before, while a smaller improvement in performance make management looks good.
- biugbkifcjk 3y agoI'm only just realizing now why I was treated so strangely after discovering a pretty severe security issue that had been in our software for about 7 years. I had only been in the team for about 12 months when I discovered it.
- throwawaaarrgh 3y agoThe only time I have seen a good response from the discovery of a large security hole was when it was by total accident, and the person mentioned it in a public channel by accident. They almost got fired, but the bug got fixed quickly.
- koliber 3y agoThe Secrets of Consulting", Gerald Weinberg advise against improving more than 10% of performance, and if so, of hoping to have any credit. Just like the article, his reasoning is that if you improve performance too much, it makes management/the team look bad for not doing it before, while a smaller improvement in performance make management looks good.
- gurchik 3y agoWhy did you copy-paste someone else's comment?
- koliber 3y agoThis was a fat-finger mistake. I copied the comment for my notes and somehow fired off a cmd-v while in the comment window. My apologies. This was not the intent.
- ThePowerOfFuet 3y agoBut then you submitted it anyway?
- koliber 3y agoYep. Not sure how it happened. I copied the note to my personal notes. I also wrote a comment. I think somewhere between the "submit" click I cmd-v'ed. Woke up surprised today to see this. I can't remove it either. I'm open for more questions.
- Shocka1 3y agoI think you are good - you just had to pass a HN interrogation first. It really delivered though. > I'm open for more questions I laughed much harder than I should of at this.
- BonoboIO 3y agoA company of maniacs, but that was a good lesson. Often it’s better to just stay silent and watch. It’s brutal to watch, but better to be the bad guy for pointing at obvious flaws. Or find and company that values your talent and knowledge.
- porygon2 3y agoReminds me of the Preventable Problem Paradox: https://medium.com/@shreyashere/why-our-leaders-fail-us-and-then-save-us-the-preventable-problem-paradox-872614ec3293 https://medium.com/@shreyashere/why-our-leaders-fail-us-and-... If you had just let the crisis occur, everyone would get a chance to spring into action like heroes, handshakes and champagne all around on a job well done.