5 ms·

Once upon a time I uncovered a bug that recovered $4MM/year in revenue. It was swept under the rug to protect the team and executives that let the blunder conti

by j0hnyl 3y ago

Once upon a time I uncovered a bug that recovered $4MM/year in revenue. It was swept under the rug to protect the team and executives that let the blunder continue for as long as it did. I didn't get a raise, but I made some allies and got to coast for a while.

chasd00 3y ago
I did something similar early in my career but it very much went noticed. The system I wrote to correct the bug was basically a MITM attack against pharmacy rx transmissions, it re-priced prescriptions just before they went to the insurance carrier's system because the pharmacies would never apply price updates (i worked at a small independent pharmacy chain, they didn't have a central dispensing system). In my infinite genius I named the system after my current crush. Corporate liked it so much they made an annual award named after my system... so therefore after my crush but i was too embarrassed to tell them the real origin of the name. That was 25 years ago but, to this day, every year a trophy gets made with the first name of my old crush printed on it and handed out. heh if she ever knew she'd be mortified.
- WD40forRust 3y ago
  You must tell her then update us on her reaction. Bonus if you record it on video. :^)
busterarm 3y ago
Very early in my career I discovered some mission-critical network devices at the hedge fund I worked at that would have caused a network loop if either of the two machines were ever rebooted. The two servers were related to some feed data and I had noticed that they were cabled together oddly based on how their function was described to me. The estimated cost of downtime for these systems was something like $7 million per minute. I had raised the issue to a couple of the staff responsible for the machines and to the networking team but was completely dismissed because "there is no way we would have hooked them up that way" and because I was the FNG. I then raised the issue again at the weekly group meeting because it seemed important -- somebody was dispatched to check visually and came back to confirm what I said. It was a big deal -- the networking team had about 2 weeks of emergency work to do to resolve the issue cleanly. EVERYONE was angry at me. Even though I had just averted a catastrophe for the company, I made everyone look bad by doing it and particularly because of my status/position on the team. It was an important lesson learned.
- tehlike 3y ago
  Keep doing the right thing.
  - busterarm 3y ago
    It's 20 years later and I've made my career off of pointing out fundamental mistakes made by very smart people. ;) (just with more skill and tact) 90% of what I do is ask dumb questions as if I'm completely clueless.
    - tehlike 3y ago
      I know the feeling. I'm similar in that regard. Debugging complex things by asking dumb questions is extremely fruitful. Also asking dumb questions to newer engs teaches them to think different aspects of a problem themselves. One of the things I'm hoping I could keep alive in my child
- phkahler 3y ago
  >> EVERYONE was angry at me. Even though I had just averted a catastrophe for the company, I made everyone look bad by doing it and particularly because of my status/position on the team. People can be excessive in both taking credit and placing blame. An appropriate and helpful way to frame this is "the system was configured incorrectly but nobody noticed because the problem never actually happened. It's a good thing the new guy had time to go through things and spotted the problem before it ever happened." No need to crucify the team or exaggerate the value of the new guy.
- mablopoule 3y ago
  On his book "The Secrets of Consulting", Gerald Weinberg advise against improving more than 10% of performance, and if so, of hoping to have any credit. Just like the article, his reasoning is that if you improve performance too much, it makes management/the team look bad for not doing it before, while a smaller improvement in performance make management looks good.
  - 3y ago

by j0hnyl 3y ago

chasd00 3y ago

I did something similar early in my career but it very much went noticed. The system I wrote to correct the bug was basically a MITM attack against pharmacy rx transmissions, it re-priced prescriptions just before they went to the insurance carrier's system because the pharmacies would never apply price updates (i worked at a small independent pharmacy chain, they didn't have a central dispensing system). In my infinite genius I named the system after my current crush. Corporate liked it so much they made an annual award named after my system... so therefore after my crush but i was too embarrassed to tell them the real origin of the name. That was 25 years ago but, to this day, every year a trophy gets made with the first name of my old crush printed on it and handed out. heh if she ever knew she'd be mortified.

WD40forRust 3y ago
You must tell her then update us on her reaction. Bonus if you record it on video. :^)

busterarm 3y ago

Very early in my career I discovered some mission-critical network devices at the hedge fund I worked at that would have caused a network loop if either of the two machines were ever rebooted. The two servers were related to some feed data and I had noticed that they were cabled together oddly based on how their function was described to me. The estimated cost of downtime for these systems was something like $7 million per minute. I had raised the issue to a couple of the staff responsible for the machines and to the networking team but was completely dismissed because "there is no way we would have hooked them up that way" and because I was the FNG. I then raised the issue again at the weekly group meeting because it seemed important -- somebody was dispatched to check visually and came back to confirm what I said. It was a big deal -- the networking team had about 2 weeks of emergency work to do to resolve the issue cleanly. EVERYONE was angry at me. Even though I had just averted a catastrophe for the company, I made everyone look bad by doing it and particularly because of my status/position on the team. It was an important lesson learned.

tehlike 3y ago
Keep doing the right thing.
- busterarm 3y ago
  It's 20 years later and I've made my career off of pointing out fundamental mistakes made by very smart people. ;) (just with more skill and tact) 90% of what I do is ask dumb questions as if I'm completely clueless.
  - tehlike 3y ago
    I know the feeling. I'm similar in that regard. Debugging complex things by asking dumb questions is extremely fruitful. Also asking dumb questions to newer engs teaches them to think different aspects of a problem themselves. One of the things I'm hoping I could keep alive in my child
phkahler 3y ago
>> EVERYONE was angry at me. Even though I had just averted a catastrophe for the company, I made everyone look bad by doing it and particularly because of my status/position on the team. People can be excessive in both taking credit and placing blame. An appropriate and helpful way to frame this is "the system was configured incorrectly but nobody noticed because the problem never actually happened. It's a good thing the new guy had time to go through things and spotted the problem before it ever happened." No need to crucify the team or exaggerate the value of the new guy.
mablopoule 3y ago
On his book "The Secrets of Consulting", Gerald Weinberg advise against improving more than 10% of performance, and if so, of hoping to have any credit. Just like the article, his reasoning is that if you improve performance too much, it makes management/the team look bad for not doing it before, while a smaller improvement in performance make management looks good.
- 3y ago